[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:46 dist-test-slave-7j43 krb5kdc[17625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:46 dist-test-slave-7j43 krb5kdc[17625](info): set up 2 sockets
Sep 27 03:17:46 dist-test-slave-7j43 krb5kdc[17625](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 03:17:47.129027 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:17:47.240707 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42825
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:47.320773 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:47.320998 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:47.321033 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:47.322245 17638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:17:47.322312 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:47.322338 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:47.322369 17638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:17:47.322393 17638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:17:47.323891 17638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:47.324155 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:47.324476 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:47.330740 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.330739 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.330971 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.331684 17638 server_base.cc:1034] running on GCE node
I20260927 03:17:47.331971 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:47.332296 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:47.333459 17638 hybrid_clock.cc:648] HybridClock initialized: now 1790479067333446 us; error 42 us; skew 500 ppm
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:47.335830 17638 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:17:47.336349 17638 webserver.cc:460] Webserver started at http://127.17.52.190:36089/ using document root <none> and password file <none>
I20260927 03:17:47.336613 17638 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:47.336692 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:47.336833 17638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:47.337944 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "ef7a49263a940e66bdd242157a5209b3"
server_key_iv: "34c0c4a47c9cf294c22d0dfa90baf5d8"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.338327 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "ef7a49263a940e66bdd242157a5209b3"
server_key_iv: "34c0c4a47c9cf294c22d0dfa90baf5d8"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.339730 17638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:17:47.340456 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.340668 17638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:47.340770 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "ef7a49263a940e66bdd242157a5209b3"
server_key_iv: "34c0c4a47c9cf294c22d0dfa90baf5d8"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.340864 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:47.352238 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:47.354184 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:47.354350 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:47.358520 17638 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:42825
I20260927 03:17:47.358552 17705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:42825 every 8 connection(s)
I20260927 03:17:47.358897 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:17:47.359722 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:47.362128 17706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090: Bootstrap starting.
I20260927 03:17:47.362761 17706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:47.363083 17706 log.cc:824] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:47.363792 17706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090: No bootstrap required, opened a new log
I20260927 03:17:47.365358 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 17638
I20260927 03:17:47.365438 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:17:47.365473 17706 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } }
I20260927 03:17:47.365640 17706 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:47.365686 17706 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57a9a1e95ec4f69b0ed1e5e25bfa090, State: Initialized, Role: FOLLOWER
I20260927 03:17:47.365854 17706 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } }
I20260927 03:17:47.365952 17706 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:17:47.365995 17706 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:17:47.366044 17706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:47.366205 17618 external_mini_cluster.cc:1428] Setting key c550630c10be244c97f8683f50782399
I20260927 03:17:47.366613 17706 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } }
I20260927 03:17:47.366730 17706 leader_election.cc:302] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57a9a1e95ec4f69b0ed1e5e25bfa090; no voters:
I20260927 03:17:47.366900 17706 leader_election.cc:288] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:17:47.366932 17709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:47.367116 17709 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [term 1 LEADER]: Becoming Leader. State: Replica: d57a9a1e95ec4f69b0ed1e5e25bfa090, State: Running, Role: LEADER
I20260927 03:17:47.367179 17706 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:17:47.367250 17709 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } }
I20260927 03:17:47.367642 17711 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57a9a1e95ec4f69b0ed1e5e25bfa090. Latest consensus state: current_term: 1 leader_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } } }
I20260927 03:17:47.367774 17711 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:47.367801 17710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57a9a1e95ec4f69b0ed1e5e25bfa090" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42825 } } }
I20260927 03:17:47.367985 17710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090 [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:47.368065 17720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:17:47.368644 17720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:17:47.370868 17720 catalog_manager.cc:1347] Generated new cluster ID: 500311a9df394fbba5ecd2e8872db46f
I20260927 03:17:47.370944 17720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:47.381664 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.367166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.17.52.190:42825)
0927 03:17:47.367336 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.367345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.367897 (+ 552us) server_negotiation.cc:407] Connection header received
0927 03:17:47.370368 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.370375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.370433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.370604 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.372181 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.373299 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.374319 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.374749 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.374871 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.378167 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.378171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.378171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.378189 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.379813 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.380245 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.380248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.380248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.380334 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.380597 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.380599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.380599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.380831 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.381004 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.381260 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.381398 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260927 03:17:47.405251 17720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:17:47.405791 17720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:17:47.411087 17720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57a9a1e95ec4f69b0ed1e5e25bfa090: Generated new TSK 0
I20260927 03:17:47.411274 17720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:17:47.437361 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:47.521670 17732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:47.521878 17732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:47.521912 17732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:47.523037 17732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:47.523102 17732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:47.523145 17732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:17:47.524648 17732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:47.524890 17732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:47.525105 17732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:47.530051 17738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.530054 17740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.530289 17737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.530263 17732 server_base.cc:1034] running on GCE node
I20260927 03:17:47.530496 17732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:47.530697 17732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:47.531875 17732 hybrid_clock.cc:648] HybridClock initialized: now 1790479067531856 us; error 36 us; skew 500 ppm
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:47.534500 17732 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:17:47.534942 17732 webserver.cc:460] Webserver started at http://127.17.52.129:33301/ using document root <none> and password file <none>
I20260927 03:17:47.535136 17732 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:47.535177 17732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:47.535264 17732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:47.536243 17732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "6451ec71cc9b46d985eab135d478a6f8"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "4758fb63e520832ad85815c969aa0880"
server_key_iv: "1a3b786baad3503585de92c84a606130"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.536536 17732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "6451ec71cc9b46d985eab135d478a6f8"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "4758fb63e520832ad85815c969aa0880"
server_key_iv: "1a3b786baad3503585de92c84a606130"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.537806 17732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:47.538427 17747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.538574 17732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:17:47.538648 17732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "6451ec71cc9b46d985eab135d478a6f8"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "4758fb63e520832ad85815c969aa0880"
server_key_iv: "1a3b786baad3503585de92c84a606130"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.538698 17732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:47.546311 17732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:47.547938 17732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:47.548043 17732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:47.548202 17732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:47.548470 17732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:47.548499 17732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.548520 17732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:47.548532 17732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.554118 17732 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:35305
I20260927 03:17:47.554179 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:35305 every 8 connection(s)
I20260927 03:17:47.554497 17732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:47.561769 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 17732
I20260927 03:17:47.561864 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:17:47.562000 17618 external_mini_cluster.cc:1428] Setting key 6d72d149cf0aa900f2723fe3438022aa
I20260927 03:17:47.564734 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.555326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:40347 (local address 127.17.52.190:42825)
0927 03:17:47.555365 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.555367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.555542 (+ 175us) server_negotiation.cc:407] Connection header received
0927 03:17:47.556125 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.556126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.556144 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.556199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.557454 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.558225 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.559461 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.559678 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.559770 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.561792 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.561793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.561793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.561801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.563278 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.563740 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.563741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.563742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.563821 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.564074 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.564075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.564075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.564211 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.564278 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.564508 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.564613 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:17:47.564785 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.555294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42825 (local address 127.17.52.129:40347)
0927 03:17:47.555423 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:47.555464 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:47.556015 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:47.556322 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:47.556326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:47.557090 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:47.557325 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.557332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.558323 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.558323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.559371 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.559374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.559774 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.559775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.559905 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.560325 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:47.560333 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:47.561690 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:47.563355 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.563361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.563363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.563663 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.563897 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.563898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.563898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.564012 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.564281 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:47.564286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:47.564388 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 03:17:47.564465 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:17:47.564621 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260927 03:17:47.565335 17861 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42825
I20260927 03:17:47.565419 17861 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:47.565594 17861 heartbeater.cc:511] Master 127.17.52.190:42825 requested a full tablet report, sending...
I20260927 03:17:47.566157 17670 ts_manager.cc:194] Registered new tserver with Master: 6451ec71cc9b46d985eab135d478a6f8 (127.17.52.129:35305)
I20260927 03:17:47.566720 17670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:40347
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:17:47.592554 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:47.672647 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:47.672837 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:47.672860 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:47.673967 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:47.674006 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:47.674031 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:17:47.675432 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:47.675659 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:47.675879 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:47.680905 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.680954 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.681008 17866 server_base.cc:1034] running on GCE node
W20260927 03:17:47.680926 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.681306 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:47.681548 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:47.682684 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790479067682661 us; error 41 us; skew 500 ppm
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:47.685215 17866 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:17:47.685659 17866 webserver.cc:460] Webserver started at http://127.17.52.130:45297/ using document root <none> and password file <none>
I20260927 03:17:47.685849 17866 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:47.685889 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:47.685974 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:47.686892 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "831a538395da4669a703da98783b050d"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "8d231785ba86fe1a7ef4a06b26946ec0"
server_key_iv: "bff1420e2d9a6fa00bed6d5f7c4469f0"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.687196 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "831a538395da4669a703da98783b050d"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "8d231785ba86fe1a7ef4a06b26946ec0"
server_key_iv: "bff1420e2d9a6fa00bed6d5f7c4469f0"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.688432 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:17:47.689131 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.689311 17866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.689430 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "831a538395da4669a703da98783b050d"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "8d231785ba86fe1a7ef4a06b26946ec0"
server_key_iv: "bff1420e2d9a6fa00bed6d5f7c4469f0"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.689510 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:47.717655 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:47.719954 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:47.720144 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:47.720367 17866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:47.720707 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:47.720767 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.720808 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:47.720837 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.726935 17866 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:39117
I20260927 03:17:47.726991 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:39117 every 8 connection(s)
I20260927 03:17:47.727317 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:17:47.727907 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 17866
I20260927 03:17:47.727977 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:17:47.728173 17618 external_mini_cluster.cc:1428] Setting key a7093daf90acd43054de8a410cbe44ea
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
I20260927 03:17:47.737952 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.728176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:39237 (local address 127.17.52.190:42825)
0927 03:17:47.728224 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.728226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.728374 (+ 148us) server_negotiation.cc:407] Connection header received
0927 03:17:47.728988 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.728989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.728999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.729035 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.730316 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.731110 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.732066 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.732391 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.732468 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.734785 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.734786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.734786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.734794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.736510 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.736953 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.736954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.736954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.737017 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.737301 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.737302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.737302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.737437 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.737493 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.737776 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.737840 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:17:47.738170 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.728059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42825 (local address 127.17.52.130:39237)
0927 03:17:47.728234 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:47.728289 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:47.728860 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:47.729121 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:47.729125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:47.729951 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:47.730227 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.730233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.731199 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.731200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.731982 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.731984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.732495 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.732496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.732672 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.733123 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:47.733131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:47.734679 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:47.736633 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.736646 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.736648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.736878 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.737106 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.737107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.737107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.737226 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.737503 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:47.737508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:47.737645 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:17:47.737736 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:17:47.737911 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260927 03:17:47.738682 17995 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42825
I20260927 03:17:47.738795 17995 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:47.738962 17995 heartbeater.cc:511] Master 127.17.52.190:42825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
I20260927 03:17:47.739405 17670 ts_manager.cc:194] Registered new tserver with Master: 831a538395da4669a703da98783b050d (127.17.52.130:39117)
I20260927 03:17:47.739785 17670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:39237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:17:47.752910 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:47.837831 18000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:47.838027 18000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:47.838050 18000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:47.839182 18000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:47.839227 18000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:47.839280 18000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:17:47.840782 18000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:47.841013 18000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:47.841243 18000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:47.846204 18008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.846315 18000 server_base.cc:1034] running on GCE node
W20260927 03:17:47.846220 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:47.846235 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:47.846699 18000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:47.846944 18000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:47.848137 18000 hybrid_clock.cc:648] HybridClock initialized: now 1790479067848127 us; error 30 us; skew 500 ppm
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:47.850704 18000 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:17:47.851146 18000 webserver.cc:460] Webserver started at http://127.17.52.131:34431/ using document root <none> and password file <none>
I20260927 03:17:47.851341 18000 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:47.851382 18000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:47.851471 18000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:47.852452 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "3c17faf2514a4c9098f32cdfa8c43494"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "b09e99d97b017329c4d79ed3c3af3ee9"
server_key_iv: "d6aaeb9d12840bee2334efc0bd77b900"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.852864 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "3c17faf2514a4c9098f32cdfa8c43494"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "b09e99d97b017329c4d79ed3c3af3ee9"
server_key_iv: "d6aaeb9d12840bee2334efc0bd77b900"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.854131 18000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:47.854866 18015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.855048 18000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:47.855146 18000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "3c17faf2514a4c9098f32cdfa8c43494"
format_stamp: "Formatted at 2026-09-27 03:17:47 on dist-test-slave-7j43"
server_key: "b09e99d97b017329c4d79ed3c3af3ee9"
server_key_iv: "d6aaeb9d12840bee2334efc0bd77b900"
server_key_version: "encryptionkey@0"
I20260927 03:17:47.855221 18000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:47.875193 18000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:47.877112 18000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:47.877271 18000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:47.877516 18000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:47.877823 18000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:47.877878 18000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.877919 18000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:47.877950 18000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:47.883632 18000 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:45835
I20260927 03:17:47.883672 18128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:45835 every 8 connection(s)
I20260927 03:17:47.884017 18000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:17:47.887683 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18000
I20260927 03:17:47.887748 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:17:47.887909 17618 external_mini_cluster.cc:1428] Setting key 9ab4b3f3512b5903eefdb4f9e98514c3
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:47.893090 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.884788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:47465 (local address 127.17.52.190:42825)
0927 03:17:47.884845 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.884847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.885053 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:17:47.885575 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.885576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.885590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.885640 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.886563 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.887073 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.888156 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.888457 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.888546 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.890699 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.890700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.890701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.890707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.891847 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.892180 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.892180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.892181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.892227 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.892531 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.892531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.892532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.892681 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.892733 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.892972 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.893029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:17:47.893290 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.884790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42825 (local address 127.17.52.131:47465)
0927 03:17:47.884935 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:47.884969 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:47.885462 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:47.885715 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:47.885719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:47.886266 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:47.886454 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.886458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.887166 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.887167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.888073 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:47.888075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.888552 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.888553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:47.888738 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.889316 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:47.889332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:47.890622 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:47.891914 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.891919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.891921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.892117 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.892285 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:47.892285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:47.892286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:47.892424 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:47.892736 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:47.892740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:47.892862 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:17:47.892944 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:17:47.893055 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 03:17:47.893719 18129 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42825
I20260927 03:17:47.893826 18129 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:47.893987 18129 heartbeater.cc:511] Master 127.17.52.190:42825 requested a full tablet report, sending...
I20260927 03:17:47.894348 17670 ts_manager.cc:194] Registered new tserver with Master: 3c17faf2514a4c9098f32cdfa8c43494 (127.17.52.131:45835)
I20260927 03:17:47.894749 17670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:47465
I20260927 03:17:47.900036 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:47.925644 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.914553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.17.52.129:35305)
0927 03:17:47.914737 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.914744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.914760 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:17:47.914796 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.914798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.914811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.914889 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.915462 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.916206 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.917014 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.917339 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.917419 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.922998 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.923013 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.923013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.923020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.924366 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.924802 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.924806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.924806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.924869 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.925079 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.925088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.925088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.925226 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.925384 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.925443 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.925521 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
W20260927 03:17:47.926034 17775 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60128
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:17:47 dist-test-slave-7j43 krb5kdc[17625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:47.947952 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:47.940858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.17.52.129:35305)
0927 03:17:47.940921 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:47.940923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:47.941047 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:17:47.941141 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:47.941142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:47.941160 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:47.941221 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:47.941782 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.942386 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.943197 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:47.943407 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:47.943492 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:47.945682 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:47.945684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:47.945684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:47.945692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:47.946853 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.947182 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.947184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.947184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.947242 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:47.947457 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:47.947458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:47.947458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:47.947554 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:47.947673 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:47.947764 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:17:47.947847 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:17:47.948506 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 17732
I20260927 03:17:47.953894 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 17866
I20260927 03:17:47.958431 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18000
I20260927 03:17:47.962559 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 17638
2026-09-27T03:17:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1866 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:48 dist-test-slave-7j43 krb5kdc[18143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:48 dist-test-slave-7j43 krb5kdc[18143](info): set up 2 sockets
Sep 27 03:17:48 dist-test-slave-7j43 krb5kdc[18143](info): commencing operation
krb5kdc: starting...
W20260927 03:17:48.984517 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:17:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:17:49.094905 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38997
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:49.173679 18156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:49.173915 18156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:49.173950 18156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:49.175060 18156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:17:49.175124 18156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:49.175151 18156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:49.175173 18156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:17:49.175201 18156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:17:49.176640 18156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:49.176911 18156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:49.177129 18156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:49.181721 18161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.181694 18164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.181713 18162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:49.181962 18156 server_base.cc:1034] running on GCE node
I20260927 03:17:49.182190 18156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:49.182428 18156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:49.183611 18156 hybrid_clock.cc:648] HybridClock initialized: now 1790479069183581 us; error 42 us; skew 500 ppm
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:49.185957 18156 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:17:49.186381 18156 webserver.cc:460] Webserver started at http://127.17.52.190:42069/ using document root <none> and password file <none>
I20260927 03:17:49.186563 18156 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:49.186601 18156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:49.186685 18156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:49.187615 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "0300c0ea45254fd491f9cf3eda80ca7f"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1c3fd3dc0f22c430830d6be6c6f69ece"
server_key_iv: "c0c2769125e6cf8290261ad6a10d0a71"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.187913 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "0300c0ea45254fd491f9cf3eda80ca7f"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1c3fd3dc0f22c430830d6be6c6f69ece"
server_key_iv: "c0c2769125e6cf8290261ad6a10d0a71"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.189190 18156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:17:49.189888 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.190065 18156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:49.190130 18156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "0300c0ea45254fd491f9cf3eda80ca7f"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1c3fd3dc0f22c430830d6be6c6f69ece"
server_key_iv: "c0c2769125e6cf8290261ad6a10d0a71"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.190181 18156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:49.210367 18156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:49.212073 18156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:49.212174 18156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:49.216001 18156 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38997
I20260927 03:17:49.216053 18223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38997 every 8 connection(s)
I20260927 03:17:49.216382 18156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:17:49.217197 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:49.219174 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18156
I20260927 03:17:49.219265 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:17:49.219424 17618 external_mini_cluster.cc:1428] Setting key 3615f9f62508ee1aa92741ccecdcb4e4
I20260927 03:17:49.219650 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f: Bootstrap starting.
I20260927 03:17:49.220340 18224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:49.220690 18224 log.cc:824] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:49.221372 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f: No bootstrap required, opened a new log
I20260927 03:17:49.223161 18224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } }
I20260927 03:17:49.223326 18224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:49.223373 18224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0300c0ea45254fd491f9cf3eda80ca7f, State: Initialized, Role: FOLLOWER
I20260927 03:17:49.223495 18224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } }
I20260927 03:17:49.223588 18224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:17:49.223631 18224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:17:49.223677 18224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:49.224187 18224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } }
I20260927 03:17:49.224310 18224 leader_election.cc:302] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0300c0ea45254fd491f9cf3eda80ca7f; no voters:
I20260927 03:17:49.224481 18224 leader_election.cc:288] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:17:49.224505 18229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:49.224637 18229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [term 1 LEADER]: Becoming Leader. State: Replica: 0300c0ea45254fd491f9cf3eda80ca7f, State: Running, Role: LEADER
I20260927 03:17:49.224718 18229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } }
I20260927 03:17:49.224756 18224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:17:49.225045 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } } }
I20260927 03:17:49.225116 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:49.225234 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0300c0ea45254fd491f9cf3eda80ca7f. Latest consensus state: current_term: 1 leader_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0300c0ea45254fd491f9cf3eda80ca7f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38997 } } }
I20260927 03:17:49.225301 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:49.225682 18233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:17:49.226140 18233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:49.230068 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.219900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.17.52.190:38997)
0927 03:17:49.220069 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.220078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.220112 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:17:49.220610 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.220619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.220684 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.220837 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.221601 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.222463 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.223208 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.223770 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.223905 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.226975 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.226979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.226980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.226992 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.228539 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.228943 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.228946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.228946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.229039 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.229241 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.229242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.229242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.229485 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.229643 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.229764 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.229901 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260927 03:17:49.231038 18233 catalog_manager.cc:1347] Generated new cluster ID: 0aba555189534fc5bfd125a2e8fc6dd2
I20260927 03:17:49.231087 18233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:17:49.239017 18233 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:17:49.239704 18233 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:17:49.245597 18233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0300c0ea45254fd491f9cf3eda80ca7f: Generated new TSK 0
I20260927 03:17:49.245788 18233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:17:49.273459 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:49.352353 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:49.352573 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:49.352634 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:49.353817 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:49.353881 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:49.353928 18250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:17:49.355436 18250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:49.355701 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:49.355937 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:49.360806 18258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.360826 18255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.360800 18256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:49.361157 18250 server_base.cc:1034] running on GCE node
I20260927 03:17:49.361375 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:49.361564 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:49.362705 18250 hybrid_clock.cc:648] HybridClock initialized: now 1790479069362693 us; error 29 us; skew 500 ppm
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:49.365144 18250 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:17:49.365592 18250 webserver.cc:460] Webserver started at http://127.17.52.129:34877/ using document root <none> and password file <none>
I20260927 03:17:49.365830 18250 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:49.365898 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:49.366043 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:49.367128 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "dc8ea225faa64dc7ab3129e080aa4ca7"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1a90046e9227c1f5352a76b9b48f2123"
server_key_iv: "d65f963b8dd43bfe406c09a3871f15c0"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.367477 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "dc8ea225faa64dc7ab3129e080aa4ca7"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1a90046e9227c1f5352a76b9b48f2123"
server_key_iv: "d65f963b8dd43bfe406c09a3871f15c0"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.368831 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:17:49.369511 18265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.369679 18250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:49.369786 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "dc8ea225faa64dc7ab3129e080aa4ca7"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "1a90046e9227c1f5352a76b9b48f2123"
server_key_iv: "d65f963b8dd43bfe406c09a3871f15c0"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.369854 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:49.382579 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:49.384390 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:49.384536 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:49.384763 18250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:49.385043 18250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:49.385075 18250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.385097 18250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:49.385144 18250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.390776 18250 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:34883
I20260927 03:17:49.390815 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:34883 every 8 connection(s)
I20260927 03:17:49.391098 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:49.398046 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18250
I20260927 03:17:49.398132 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:17:49.398306 17618 external_mini_cluster.cc:1428] Setting key 30ba2e44b80debdf1f005c939ea50b09
I20260927 03:17:49.400849 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.391733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:55605 (local address 127.17.52.190:38997)
0927 03:17:49.391814 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.391816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.391990 (+ 174us) server_negotiation.cc:407] Connection header received
0927 03:17:49.392970 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.392971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.392982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.393033 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.393564 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.394141 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.395096 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.395303 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.395366 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.397638 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.397639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.397640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.397647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.399282 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.399756 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.399757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.399757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.399817 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.400080 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.400081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.400081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.400194 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.400258 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.400622 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.400714 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:17:49.400880 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.391715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38997 (local address 127.17.52.129:55605)
0927 03:17:49.391876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.391911 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.392864 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.393084 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.393087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.393318 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:49.393492 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.393496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.394215 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.394216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.395000 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.395002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.395380 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.395381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.395472 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.395906 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:49.395913 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:49.397539 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:49.399360 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.399366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.399369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.399682 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.399878 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.399879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.399879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.400019 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.400269 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:49.400276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:49.400444 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.400548 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.400727 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 03:17:49.401343 18379 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38997
I20260927 03:17:49.401435 18379 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:49.401578 18379 heartbeater.cc:511] Master 127.17.52.190:38997 requested a full tablet report, sending...
I20260927 03:17:49.402024 18188 ts_manager.cc:194] Registered new tserver with Master: dc8ea225faa64dc7ab3129e080aa4ca7 (127.17.52.129:34883)
I20260927 03:17:49.402631 18188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:55605
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:17:49.421007 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:49.500034 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:49.500236 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:49.500258 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:49.501323 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:49.501358 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:49.501384 18384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:17:49.502733 18384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:49.502902 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:49.503054 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:49.507688 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.507790 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:49.507859 18384 server_base.cc:1034] running on GCE node
W20260927 03:17:49.507688 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:49.508119 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:49.508306 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:49.509469 18384 hybrid_clock.cc:648] HybridClock initialized: now 1790479069509449 us; error 43 us; skew 500 ppm
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:49.511749 18384 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:17:49.512203 18384 webserver.cc:460] Webserver started at http://127.17.52.130:39223/ using document root <none> and password file <none>
I20260927 03:17:49.512423 18384 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:49.512466 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:49.512625 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:49.513597 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "9478fd70798737eb825d15cc31ce0598"
server_key_iv: "8bd0d828e4867f7ec1c89735940c8b9d"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.513944 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "9478fd70798737eb825d15cc31ce0598"
server_key_iv: "8bd0d828e4867f7ec1c89735940c8b9d"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.515166 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:17:49.515823 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.515993 18384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:49.516085 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "9478fd70798737eb825d15cc31ce0598"
server_key_iv: "8bd0d828e4867f7ec1c89735940c8b9d"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.516167 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:49.544221 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:49.546068 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:49.546226 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:49.546444 18384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:49.546751 18384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:49.546818 18384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.546850 18384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:49.546877 18384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.552529 18384 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:37579
I20260927 03:17:49.552598 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:37579 every 8 connection(s)
I20260927 03:17:49.552886 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:17:49.555351 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18384
I20260927 03:17:49.555459 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:17:49.555644 17618 external_mini_cluster.cc:1428] Setting key be52d75a53ad1dc1a8773fe61be42fb2
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:49.562983 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.553617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:48231 (local address 127.17.52.190:38997)
0927 03:17:49.553685 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.553687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.553904 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:17:49.554813 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.554814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.554826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.554872 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.555462 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.556122 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.557246 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.557464 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.557543 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.560049 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.560053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.560053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.560064 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.561502 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.562039 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.562040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.562040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.562086 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.562367 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.562368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.562368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.562461 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.562508 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.562859 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.562922 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:17:49.563285 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.553629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38997 (local address 127.17.52.130:48231)
0927 03:17:49.553778 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.553813 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.554696 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.554940 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.554943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.555177 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:49.555354 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.555359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.556209 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.556210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.557152 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.557154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.557545 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.557546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.557668 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.558207 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:49.558215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:49.559942 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:49.561593 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.561600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.561603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.561956 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.562150 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.562151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.562151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.562295 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.562527 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:49.562534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:49.562714 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.562842 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.563022 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260927 03:17:49.563925 18513 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38997
I20260927 03:17:49.564045 18513 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:49.564225 18513 heartbeater.cc:511] Master 127.17.52.190:38997 requested a full tablet report, sending...
I20260927 03:17:49.564764 18188 ts_manager.cc:194] Registered new tserver with Master: 2ad6a2aa4cda42f0aca0f8c8f04cde1d (127.17.52.130:37579)
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
I20260927 03:17:49.565274 18188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:48231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:17:49.580705 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:49.661439 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:49.661626 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:49.661649 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:49.662710 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:49.662743 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:49.662770 18518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:17:49.664191 18518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:49.664376 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:49.664534 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:49.669579 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.669615 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:49.669632 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:49.669839 18518 server_base.cc:1034] running on GCE node
I20260927 03:17:49.670133 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:49.670444 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:49.671612 18518 hybrid_clock.cc:648] HybridClock initialized: now 1790479069671588 us; error 31 us; skew 500 ppm
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:49.675083 18518 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:17:49.675592 18518 webserver.cc:460] Webserver started at http://127.17.52.131:43007/ using document root <none> and password file <none>
I20260927 03:17:49.675869 18518 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:49.675916 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:49.676004 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:49.677037 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "f77194a0a2b74934807c20ffc6af452d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "0c3aa0c047cbd85d241ee38213d0f854"
server_key_iv: "6902e280997d7d53784b1e9e2f0f8290"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.677335 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "f77194a0a2b74934807c20ffc6af452d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "0c3aa0c047cbd85d241ee38213d0f854"
server_key_iv: "6902e280997d7d53784b1e9e2f0f8290"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.678611 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:17:49.679244 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.679392 18518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:49.679472 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "f77194a0a2b74934807c20ffc6af452d"
format_stamp: "Formatted at 2026-09-27 03:17:49 on dist-test-slave-7j43"
server_key: "0c3aa0c047cbd85d241ee38213d0f854"
server_key_iv: "6902e280997d7d53784b1e9e2f0f8290"
server_key_version: "encryptionkey@0"
I20260927 03:17:49.679523 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:49.701680 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:49.703501 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:49.703611 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:49.703800 18518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:49.704119 18518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:49.704152 18518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.704195 18518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:49.704214 18518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:49.710043 18518 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:38301
I20260927 03:17:49.710119 18646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:38301 every 8 connection(s)
I20260927 03:17:49.710424 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:17:49.714692 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18518
I20260927 03:17:49.714804 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:17:49.714982 17618 external_mini_cluster.cc:1428] Setting key 26108aea6de1f2770e34c9a839fad27e
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:49.719551 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.711140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:50615 (local address 127.17.52.190:38997)
0927 03:17:49.711203 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.711205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.711419 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:17:49.712314 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.712315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.712324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.712378 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.713044 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.713564 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.714524 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.714701 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.714774 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.717083 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.717084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.717084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.717090 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.718456 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.718829 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.718830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.718831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.718879 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.719048 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.719048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.719049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.719142 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.719192 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.719431 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.719489 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:17:49.719803 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.711132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38997 (local address 127.17.52.131:50615)
0927 03:17:49.711295 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.711336 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.712225 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.712448 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.712451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.712774 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:49.712962 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.712967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.713636 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.713636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.714452 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.714453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.714792 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.714793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.714944 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.715497 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:49.715507 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:49.717006 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:49.718518 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.718522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.718523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.718768 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.718921 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:49.718921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:49.718921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:49.718999 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:49.719195 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:49.719198 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:49.719324 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.719407 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.719518 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 03:17:49.720273 18647 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38997
I20260927 03:17:49.720364 18647 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:49.720526 18647 heartbeater.cc:511] Master 127.17.52.190:38997 requested a full tablet report, sending...
I20260927 03:17:49.720947 18188 ts_manager.cc:194] Registered new tserver with Master: f77194a0a2b74934807c20ffc6af452d (127.17.52.131:38301)
I20260927 03:17:49.721369 18188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:50615
I20260927 03:17:49.726656 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:17:49.733608 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.728181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.17.52.190:38997)
0927 03:17:49.728265 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.728267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.728318 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:17:49.728354 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.728354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.728362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.728407 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.728983 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.729445 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.730129 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.730310 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.730360 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.731461 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.731462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.731462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.731467 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.732680 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.732963 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.732963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.732964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.733008 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.733189 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.733189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.733189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.733308 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.733379 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.733477 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.733530 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 03:17:49.735572 18188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:49.737337 18188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:49.743098 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38301 (local address 127.0.0.1:56648)
0927 03:17:49.739771 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.739782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.739834 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.740114 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.740115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.740117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:49.740168 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.740170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741144 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.741145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.741929 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.741931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.742750 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.742751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.742828 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.742858 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.742966 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.743013 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":49,"threads_started":1}
I20260927 03:17:49.743108 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.17.52.131:38301)
0927 03:17:49.739971 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.739976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.739986 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:49.740022 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.740024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.740037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.740111 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:49.740222 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741049 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.742029 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.742669 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.742751 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.742813 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.742980 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.743018 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":99,"threads_started":1}
I20260927 03:17:49.743886 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:34883 (local address 127.0.0.1:55612)
0927 03:17:49.739719 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.739755 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.739818 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.740016 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.740018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.740021 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:49.740086 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.740088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741560 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.741561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.742776 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.742778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.743654 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.743655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.743715 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.743718 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.743772 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.743811 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":73,"thread_start_us":57,"threads_started":1}
I20260927 03:17:49.743896 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.17.52.129:34883)
0927 03:17:49.739793 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.739800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.739810 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:49.739899 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.739901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.739915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.740012 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:49.740181 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741435 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.742925 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.743554 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.743649 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.743700 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.743793 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.743826 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260927 03:17:49.743795 18581 tablet_service.cc:1467] Processing CreateTablet for tablet a19971584eb54d76adbaf5a3de8d4e4b (DEFAULT_TABLE table=test-table [id=018201f0ae904773bcc2a1cdb2c798dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:49.744155 18581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19971584eb54d76adbaf5a3de8d4e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:49.744647 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37579 (local address 127.0.0.1:34662)
0927 03:17:49.739921 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.739930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.739967 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.740281 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.740281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.740282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:49.740309 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.740310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741562 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.741562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.742779 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.742781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.744384 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.744385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.744443 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.744447 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.744496 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.744540 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":49,"thread_start_us":31,"threads_started":1}
I20260927 03:17:49.744499 18313 tablet_service.cc:1467] Processing CreateTablet for tablet a19971584eb54d76adbaf5a3de8d4e4b (DEFAULT_TABLE table=test-table [id=018201f0ae904773bcc2a1cdb2c798dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:49.744868 18313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19971584eb54d76adbaf5a3de8d4e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:49.745029 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.739750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.17.52.130:37579)
0927 03:17:49.740140 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.740144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.740154 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:49.740189 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.740190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.740205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.740273 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:49.740351 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.741451 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.742888 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.743925 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.744292 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.744349 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.744877 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.744957 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":102,"threads_started":1}
I20260927 03:17:49.745913 18447 tablet_service.cc:1467] Processing CreateTablet for tablet a19971584eb54d76adbaf5a3de8d4e4b (DEFAULT_TABLE table=test-table [id=018201f0ae904773bcc2a1cdb2c798dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:49.746219 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19971584eb54d76adbaf5a3de8d4e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:49.747623 18661 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: Bootstrap starting.
I20260927 03:17:49.748126 18661 tablet_bootstrap.cc:654] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:49.748720 18661 log.cc:824] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:49.748785 18662 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: Bootstrap starting.
I20260927 03:17:49.749377 18662 tablet_bootstrap.cc:654] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:49.749480 18661 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: No bootstrap required, opened a new log
I20260927 03:17:49.749558 18661 ts_tablet_manager.cc:1397] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:17:49.749846 18662 log.cc:824] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:49.750147 18663 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Bootstrap starting.
I20260927 03:17:49.750540 18662 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: No bootstrap required, opened a new log
I20260927 03:17:49.750607 18662 ts_tablet_manager.cc:1397] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:17:49.750775 18663 tablet_bootstrap.cc:654] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:49.751283 18663 log.cc:824] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:49.751230 18661 raft_consensus.cc:348] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.751385 18661 raft_consensus.cc:374] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:49.751425 18661 raft_consensus.cc:729] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ea225faa64dc7ab3129e080aa4ca7, State: Initialized, Role: FOLLOWER
I20260927 03:17:49.751535 18661 consensus_queue.cc:260] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.751821 18661 ts_tablet_manager.cc:1428] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:17:49.751945 18663 tablet_bootstrap.cc:492] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: No bootstrap required, opened a new log
I20260927 03:17:49.751958 18379 heartbeater.cc:503] Master 127.17.52.190:38997 was elected leader, sending a full tablet report...
I20260927 03:17:49.752012 18663 ts_tablet_manager.cc:1397] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:17:49.752718 18662 raft_consensus.cc:348] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.752900 18662 raft_consensus.cc:374] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:49.752936 18662 raft_consensus.cc:729] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77194a0a2b74934807c20ffc6af452d, State: Initialized, Role: FOLLOWER
I20260927 03:17:49.753046 18662 consensus_queue.cc:260] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.753304 18662 ts_tablet_manager.cc:1428] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:17:49.753314 18647 heartbeater.cc:503] Master 127.17.52.190:38997 was elected leader, sending a full tablet report...
I20260927 03:17:49.753845 18663 raft_consensus.cc:348] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.753991 18663 raft_consensus.cc:374] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:49.754042 18663 raft_consensus.cc:729] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad6a2aa4cda42f0aca0f8c8f04cde1d, State: Initialized, Role: FOLLOWER
I20260927 03:17:49.754150 18663 consensus_queue.cc:260] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.754406 18663 ts_tablet_manager.cc:1428] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:17:49.754423 18513 heartbeater.cc:503] Master 127.17.52.190:38997 was elected leader, sending a full tablet report...
I20260927 03:17:49.797025 18667 raft_consensus.cc:482] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:49.797138 18667 raft_consensus.cc:504] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.797433 18667 leader_election.cc:288] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f77194a0a2b74934807c20ffc6af452d (127.17.52.131:38301), 2ad6a2aa4cda42f0aca0f8c8f04cde1d (127.17.52.130:37579)
I20260927 03:17:49.800279 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.797691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:45049 (local address 127.17.52.130:37579)
0927 03:17:49.797734 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.797736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.797747 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:17:49.797774 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.797775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.797785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.797858 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:49.798034 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.798594 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.799566 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.799942 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.799996 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.800003 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.800167 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.800209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:17:49.800274 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.797595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37579 (local address 127.17.52.129:45049)
0927 03:17:49.797637 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.797650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.797706 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.797875 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.797876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.797878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:49.797975 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.797977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.798669 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.798670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.799484 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.799486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.800019 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.800020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.800100 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.800103 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.800150 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.800179 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":31}
I20260927 03:17:49.800379 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.797663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:42493 (local address 127.17.52.131:38301)
0927 03:17:49.797712 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.797714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.797965 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:17:49.797993 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.797994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.798005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.798060 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:49.798214 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.798689 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.799612 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.799996 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.800067 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.800075 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.800260 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.800306 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:17:49.800400 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.797596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38301 (local address 127.17.52.129:42493)
0927 03:17:49.797885 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:49.797894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:49.797954 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:49.798080 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:49.798081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:49.798082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:49.798149 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.798151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.798771 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.798772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.799500 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:49.799502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.800092 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.800093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:49.800174 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.800179 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:17:49.800232 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:17:49.800276 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":116,"thread_start_us":125,"threads_started":1}
I20260927 03:17:49.800668 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19971584eb54d76adbaf5a3de8d4e4b" candidate_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" is_pre_election: true
I20260927 03:17:49.800748 18601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19971584eb54d76adbaf5a3de8d4e4b" candidate_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77194a0a2b74934807c20ffc6af452d" is_pre_election: true
I20260927 03:17:49.800848 18467 raft_consensus.cc:2454] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8ea225faa64dc7ab3129e080aa4ca7 in term 0.
I20260927 03:17:49.800905 18601 raft_consensus.cc:2454] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8ea225faa64dc7ab3129e080aa4ca7 in term 0.
I20260927 03:17:49.801122 18269 leader_election.cc:302] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ea225faa64dc7ab3129e080aa4ca7, f77194a0a2b74934807c20ffc6af452d; no voters:
I20260927 03:17:49.801281 18667 raft_consensus.cc:2781] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:49.801332 18667 raft_consensus.cc:482] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:49.801348 18667 raft_consensus.cc:3037] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:49.802035 18667 raft_consensus.cc:504] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.802169 18667 leader_election.cc:288] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [CANDIDATE]: Term 1 election: Requested vote from peers f77194a0a2b74934807c20ffc6af452d (127.17.52.131:38301), 2ad6a2aa4cda42f0aca0f8c8f04cde1d (127.17.52.130:37579)
I20260927 03:17:49.802337 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19971584eb54d76adbaf5a3de8d4e4b" candidate_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d"
I20260927 03:17:49.802345 18601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19971584eb54d76adbaf5a3de8d4e4b" candidate_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77194a0a2b74934807c20ffc6af452d"
I20260927 03:17:49.802412 18467 raft_consensus.cc:3037] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:49.802410 18601 raft_consensus.cc:3037] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:49.803258 18467 raft_consensus.cc:2454] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc8ea225faa64dc7ab3129e080aa4ca7 in term 1.
I20260927 03:17:49.803267 18601 raft_consensus.cc:2454] T a19971584eb54d76adbaf5a3de8d4e4b P f77194a0a2b74934807c20ffc6af452d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc8ea225faa64dc7ab3129e080aa4ca7 in term 1.
I20260927 03:17:49.803401 18269 leader_election.cc:302] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ea225faa64dc7ab3129e080aa4ca7, f77194a0a2b74934807c20ffc6af452d; no voters:
W20260927 03:17:49.803391 18514 tablet.cc:2369] T a19971584eb54d76adbaf5a3de8d4e4b P 2ad6a2aa4cda42f0aca0f8c8f04cde1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:49.803485 18667 raft_consensus.cc:2781] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:49.803635 18667 raft_consensus.cc:686] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [term 1 LEADER]: Becoming Leader. State: Replica: dc8ea225faa64dc7ab3129e080aa4ca7, State: Running, Role: LEADER
I20260927 03:17:49.803751 18667 consensus_queue.cc:237] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } }
I20260927 03:17:49.804509 18188 catalog_manager.cc:5495] T a19971584eb54d76adbaf5a3de8d4e4b P dc8ea225faa64dc7ab3129e080aa4ca7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc8ea225faa64dc7ab3129e080aa4ca7 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ea225faa64dc7ab3129e080aa4ca7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f77194a0a2b74934807c20ffc6af452d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad6a2aa4cda42f0aca0f8c8f04cde1d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37579 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:49.819018 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.812066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.17.52.129:34883)
0927 03:17:49.812137 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.812139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.812239 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:17:49.812317 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.812317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.812327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.812394 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.812988 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.813419 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.814102 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.814287 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.814355 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.816525 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.816532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.816533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.816540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.817871 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.818218 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.818221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.818221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.818278 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.818499 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.818500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.818500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.818623 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.818812 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.818848 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.818922 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:49.840714 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.834241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.17.52.129:34883)
0927 03:17:49.834305 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.834308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.834391 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:17:49.834470 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.834470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.834486 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.834549 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.835081 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.835549 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.836277 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.836469 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.836552 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.838653 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.838654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.838655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.838661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.839770 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.840055 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.840056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.840056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.840100 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.840248 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.840249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.840249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.840338 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.840428 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.840540 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.840654 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:49.860613 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:49.854490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.17.52.129:34883)
0927 03:17:49.854547 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:49.854549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:49.854638 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:17:49.854681 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:49.854682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:49.854690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:49.854737 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:49.855218 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.855673 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.856331 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:49.856498 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:49.856544 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:49.858604 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:49.858605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:49.858606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:49.858612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:49.859731 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.860017 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.860018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.860018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.860063 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:49.860212 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:49.860213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:49.860213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:49.860302 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:49.860373 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:49.860453 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:17:49.860506 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 03:17:49.861153 18293 tablet_service.cc:2908] Rejecting scan request for tablet a19971584eb54d76adbaf5a3de8d4e4b: Uninitialized: clean time has not yet been initialized
I20260927 03:17:49.861912 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18250
I20260927 03:17:49.866572 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18384
I20260927 03:17:49.871774 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18518
I20260927 03:17:49.876308 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18156
2026-09-27T03:17:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1913 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18685](info): set up 2 sockets
Sep 27 03:17:49 dist-test-slave-7j43 krb5kdc[18685](info): commencing operation
krb5kdc: starting...
W20260927 03:17:50.900533 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:17:50 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:17:51.009601 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38281
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:51.088752 18698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:51.088970 18698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:51.089008 18698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:51.090152 18698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:17:51.090219 18698 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:17:51.090245 18698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:51.090278 18698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:51.090302 18698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:17:51.090332 18698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:17:51.091809 18698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38281
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:51.092067 18698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:51.092293 18698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:17:51.097054 18698 server_base.cc:1034] running on GCE node
W20260927 03:17:51.097018 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.097018 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.097025 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.097623 18698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:51.097918 18698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:51.099083 18698 hybrid_clock.cc:648] HybridClock initialized: now 1790479071099070 us; error 40 us; skew 500 ppm
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:51.101513 18698 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:17:51.101975 18698 webserver.cc:460] Webserver started at http://127.17.52.190:36107/ using document root <none> and password file <none>
I20260927 03:17:51.102213 18698 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:51.102289 18698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:51.102447 18698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:51.103451 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "6da0e55de8c146c3add652d407837bef"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "d5b15c40dae9530e40bea490285b65af"
server_key_iv: "c4f437d816449a21399771a418bc7f31"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.103806 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "6da0e55de8c146c3add652d407837bef"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "d5b15c40dae9530e40bea490285b65af"
server_key_iv: "c4f437d816449a21399771a418bc7f31"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.105216 18698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:17:51.105940 18714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.106112 18698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:51.106207 18698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "6da0e55de8c146c3add652d407837bef"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "d5b15c40dae9530e40bea490285b65af"
server_key_iv: "c4f437d816449a21399771a418bc7f31"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.106302 18698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:51.126183 18698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:51.127848 18698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:51.128008 18698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:51.132678 18698 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38281
I20260927 03:17:51.132735 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38281 every 8 connection(s)
I20260927 03:17:51.133033 18698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:17:51.133592 18778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.133920 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18698
I20260927 03:17:51.134008 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:17:51.134174 17618 external_mini_cluster.cc:1428] Setting key ff9b766af0c379246a948eba02714f85
I20260927 03:17:51.136466 18778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Bootstrap starting.
I20260927 03:17:51.137255 18778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.137579 18778 log.cc:824] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:51.138283 18778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: No bootstrap required, opened a new log
I20260927 03:17:51.140906 18778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } }
I20260927 03:17:51.141031 18778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.141052 18778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da0e55de8c146c3add652d407837bef, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.141180 18778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } }
I20260927 03:17:51.141249 18778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:17:51.141273 18778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.141332 18778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.141901 18778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } }
I20260927 03:17:51.142009 18778 leader_election.cc:302] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6da0e55de8c146c3add652d407837bef; no voters:
I20260927 03:17:51.142143 18778 leader_election.cc:288] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:17:51.142263 18783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:51.142418 18778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:17:51.142607 18783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [term 1 LEADER]: Becoming Leader. State: Replica: 6da0e55de8c146c3add652d407837bef, State: Running, Role: LEADER
I20260927 03:17:51.142766 18783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } }
I20260927 03:17:51.143173 18784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6da0e55de8c146c3add652d407837bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } } }
I20260927 03:17:51.143260 18784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:51.143419 18785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6da0e55de8c146c3add652d407837bef. Latest consensus state: current_term: 1 leader_uuid: "6da0e55de8c146c3add652d407837bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da0e55de8c146c3add652d407837bef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38281 } } }
I20260927 03:17:51.143509 18785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:51.143482 18793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:17:51.144171 18793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:17:51.145771 18793 catalog_manager.cc:1347] Generated new cluster ID: 8c3bc87404d847ffb3dffec1fa4832cd
I20260927 03:17:51.145843 18793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:17:51.145991 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.134718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.17.52.190:38281)
0927 03:17:51.134884 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.134891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.134915 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:17:51.135308 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.135314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.135358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.135469 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.136205 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.137608 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.138637 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.139019 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.139097 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.141778 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.141783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.141783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.141794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.144246 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.144674 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.144677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.144677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.144798 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.145002 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.145003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.145003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.145306 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.145453 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.145547 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.145682 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260927 03:17:51.165506 18793 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:17:51.166046 18793 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:17:51.173157 18793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Generated new TSK 0
I20260927 03:17:51.173373 18793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:17:51.200376 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38281
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:51.294818 18804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:51.295042 18804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:51.295085 18804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:51.296228 18804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:51.296260 18804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:51.296280 18804 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:51.296298 18804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:17:51.297706 18804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:51.297891 18804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:51.298062 18804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:17:51.302922 18804 server_base.cc:1034] running on GCE node
W20260927 03:17:51.302910 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.302882 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.302875 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.303309 18804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:51.303544 18804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:51.304700 18804 hybrid_clock.cc:648] HybridClock initialized: now 1790479071304677 us; error 33 us; skew 500 ppm
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:51.307067 18804 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:17:51.307422 18804 webserver.cc:460] Webserver started at http://127.17.52.129:35897/ using document root <none> and password file <none>
I20260927 03:17:51.307597 18804 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:51.307636 18804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:51.307720 18804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:51.308729 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "409b9af6c9c14384b10e8c73de5ac76f"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "208a888898adc70131aa8e8fc3ec0437"
server_key_iv: "6862c913171887baafdb0b60aa9ea327"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.309154 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "409b9af6c9c14384b10e8c73de5ac76f"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "208a888898adc70131aa8e8fc3ec0437"
server_key_iv: "6862c913171887baafdb0b60aa9ea327"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.310446 18804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:17:51.311093 18820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.311275 18804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:17:51.311360 18804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "409b9af6c9c14384b10e8c73de5ac76f"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "208a888898adc70131aa8e8fc3ec0437"
server_key_iv: "6862c913171887baafdb0b60aa9ea327"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.311431 18804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:51.323612 18804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:51.325436 18804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:51.325587 18804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:51.326064 18804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:51.326100 18804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.326146 18804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:51.326175 18804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.333570 18804 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:36207
I20260927 03:17:51.333657 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:36207 every 8 connection(s)
I20260927 03:17:51.333943 18804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:17:51.335683 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18804
I20260927 03:17:51.335778 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:17:51.335948 17618 external_mini_cluster.cc:1428] Setting key 0aa0a2a2b287ed2b1b80a4a5e9c62e1d
I20260927 03:17:51.336577 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.326680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:38475 (local address 127.17.52.190:38281)
0927 03:17:51.326751 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.326753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.326928 (+ 175us) server_negotiation.cc:407] Connection header received
0927 03:17:51.327375 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.327377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.327391 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.327466 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.328101 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.328955 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.330139 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.330366 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.330450 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.332982 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.332984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.332984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.332992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.334650 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.335070 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.335071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.335072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.335125 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.335377 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.335378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.335378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.335495 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.335571 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.336380 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.336452 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:17:51.336777 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.326637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.129:38475)
0927 03:17:51.326799 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.326839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.327287 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.327551 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.327555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.327840 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.328025 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.328030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.329047 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.329048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.330032 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.330034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.330461 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.330462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.330589 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.331092 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.331103 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.332886 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.334756 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.334768 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.334770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.334983 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.335197 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.335198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.335198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.335309 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.336114 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.336119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.336227 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.336363 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.336518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260927 03:17:51.337323 18936 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38281
I20260927 03:17:51.337412 18936 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:51.337589 18936 heartbeater.cc:511] Master 127.17.52.190:38281 requested a full tablet report, sending...
I20260927 03:17:51.338145 18731 ts_manager.cc:194] Registered new tserver with Master: 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129:36207)
I20260927 03:17:51.339052 18731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:38475
I20260927 03:17:51.344228 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.338423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:49165 (local address 127.17.52.190:38281)
0927 03:17:51.338463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.338465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.338577 (+ 112us) server_negotiation.cc:407] Connection header received
0927 03:17:51.338646 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.338647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.338657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.338711 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.339022 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.339750 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.340437 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.340721 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.340809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.341659 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.341660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.341661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.341668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.343112 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.343465 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.343466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.343467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.343521 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.343733 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.343734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.343735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.343843 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.343917 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.344066 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.344140 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:17:51.344261 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.338380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.129:49165)
0927 03:17:51.338506 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.338515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.338590 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.338715 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.338716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.338890 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.338942 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.338943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.339861 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.339861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.340352 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.340353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.340811 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.340812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.340884 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.341321 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.341324 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.341574 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.343187 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.343188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.343188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.343365 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.343606 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.343607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.343607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.343669 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.343909 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.343910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.343954 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.344041 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.344170 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:17:51.360556 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38281
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:51.439240 18945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:51.439430 18945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:51.439452 18945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:51.440512 18945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:51.440543 18945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:51.440563 18945 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:51.440596 18945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:17:51.442027 18945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:51.442239 18945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:51.442457 18945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:51.447466 18954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.447448 18950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.447474 18945 server_base.cc:1034] running on GCE node
W20260927 03:17:51.447436 18952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.447851 18945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:51.448074 18945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:51.449218 18945 hybrid_clock.cc:648] HybridClock initialized: now 1790479071449203 us; error 29 us; skew 500 ppm
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:51.451574 18945 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:17:51.451989 18945 webserver.cc:460] Webserver started at http://127.17.52.130:36681/ using document root <none> and password file <none>
I20260927 03:17:51.452205 18945 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:51.452272 18945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:51.452425 18945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:51.453495 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "fcce5472c6ca4971a49c02312c218161"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "4804bacb023e3aa31200e3e594e1c4aa"
server_key_iv: "180719fdb3c3bce048ab424e7669e3f6"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.453855 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "fcce5472c6ca4971a49c02312c218161"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "4804bacb023e3aa31200e3e594e1c4aa"
server_key_iv: "180719fdb3c3bce048ab424e7669e3f6"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.455099 18945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:17:51.455798 18961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.455962 18945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:51.456058 18945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "fcce5472c6ca4971a49c02312c218161"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "4804bacb023e3aa31200e3e594e1c4aa"
server_key_iv: "180719fdb3c3bce048ab424e7669e3f6"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.456147 18945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:51.466398 18945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:51.468041 18945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:51.468195 18945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:51.468700 18945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:51.468766 18945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.468807 18945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:51.468839 18945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.475256 18945 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:44509
I20260927 03:17:51.475318 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:44509 every 8 connection(s)
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.475596 18945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:17:51.478426 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.469109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:49231 (local address 127.17.52.190:38281)
0927 03:17:51.469165 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.469167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.469374 (+ 207us) server_negotiation.cc:407] Connection header received
0927 03:17:51.469871 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.469873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.469886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.469937 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.470733 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.471597 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.473152 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.473473 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.473558 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.476077 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.476079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.476079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.476087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.477399 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.477736 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.477737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.477737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.477788 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.477982 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.477983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.477983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.478076 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.478124 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.478314 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.478366 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:17:51.478595 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.469099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.130:49231)
0927 03:17:51.469239 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.469274 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.469746 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.469998 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.470002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.470333 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.470606 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.470612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.471697 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.471698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.473062 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.473064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.473560 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.473561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.473655 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.474175 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.474187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.475995 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.477468 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.477473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.477474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.477673 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.477843 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.477844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.477844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.477929 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.478127 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.478130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.478227 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.478304 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.478411 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260927 03:17:51.479066 19077 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38281
I20260927 03:17:51.479143 19077 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:51.479322 19077 heartbeater.cc:511] Master 127.17.52.190:38281 requested a full tablet report, sending...
I20260927 03:17:51.479665 18732 ts_manager.cc:194] Registered new tserver with Master: fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509)
I20260927 03:17:51.480168 18732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:49231
I20260927 03:17:51.484724 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.480059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:57007 (local address 127.17.52.190:38281)
0927 03:17:51.480098 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.480100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.480187 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:17:51.480262 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.480263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.480274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.480330 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.480690 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.481119 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.481735 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.481902 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.481948 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.482773 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.482773 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.482774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.482778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.483823 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.484098 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.484099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.484099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.484144 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.484291 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.484291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.484291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.484405 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.484453 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.484553 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.484656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:17:51.484732 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.479994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.130:57007)
0927 03:17:51.480117 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.480128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.480185 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.480334 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.480335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.480537 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.480626 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.480628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.481203 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.481204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.481670 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.481671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.481959 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.481960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.482010 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.482417 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.482422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.482706 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.483887 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.483888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.483888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.484039 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.484195 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.484195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.484195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.484240 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.484455 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.484455 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.484488 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.484536 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.484657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":119,"threads_started":1}
I20260927 03:17:51.485040 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 18945
I20260927 03:17:51.485124 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:17:51.485321 17618 external_mini_cluster.cc:1428] Setting key 622e90e128141089382ac9cfbecbee80
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:17:51.507046 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38281
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:51.588546 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:51.588850 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:51.588886 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:51.590652 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:51.590706 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:51.590739 19087 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:51.590775 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:17:51.592160 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:51.592399 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:51.592691 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:51.599277 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:51.599303 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.599634 19087 server_base.cc:1034] running on GCE node
W20260927 03:17:51.599905 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:51.600162 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:51.600472 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:51.601624 19087 hybrid_clock.cc:648] HybridClock initialized: now 1790479071601605 us; error 33 us; skew 500 ppm
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:51.603991 19087 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:17:51.604400 19087 webserver.cc:460] Webserver started at http://127.17.52.131:39079/ using document root <none> and password file <none>
I20260927 03:17:51.604681 19087 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:51.604748 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:51.604928 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:51.605952 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "61ece68ca19b41d987f1846af4ee8ba0"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "fef8e49f0c33f6a6dc3360e7b601deb0"
server_key_iv: "712f90f5f0364de85d3508d02cbe2b45"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.606307 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "61ece68ca19b41d987f1846af4ee8ba0"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "fef8e49f0c33f6a6dc3360e7b601deb0"
server_key_iv: "712f90f5f0364de85d3508d02cbe2b45"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.607606 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:17:51.608312 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.608510 19087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:17:51.608631 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "61ece68ca19b41d987f1846af4ee8ba0"
format_stamp: "Formatted at 2026-09-27 03:17:51 on dist-test-slave-7j43"
server_key: "fef8e49f0c33f6a6dc3360e7b601deb0"
server_key_iv: "712f90f5f0364de85d3508d02cbe2b45"
server_key_version: "encryptionkey@0"
I20260927 03:17:51.608701 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:51.625679 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:51.627321 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:51.627475 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:51.627956 19087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:51.628014 19087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.628048 19087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:51.628091 19087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.635504 19087 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:33519
I20260927 03:17:51.635819 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:17:51.635959 19218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:33519 every 8 connection(s)
I20260927 03:17:51.637795 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.628624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:58259 (local address 127.17.52.190:38281)
0927 03:17:51.628668 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.628670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.628902 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:17:51.629359 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.629360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.629375 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.629449 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.630066 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.630901 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.631973 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.632165 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.632252 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.635049 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.635050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.635050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.635059 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.636673 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.637004 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.637004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.637005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.637055 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.637228 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.637228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.637228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.637337 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.637396 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.637665 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.637724 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:17:51.638006 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.628552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.131:58259)
0927 03:17:51.628742 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.628805 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.629267 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.629535 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.629538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.629820 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.629990 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.629995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.630996 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.630997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.631866 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.631868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.632282 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.632283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.632405 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.632943 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.632951 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.634956 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.636738 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.636742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.636743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.636941 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.637098 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.637098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.637099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.637181 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.637402 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.637406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.637554 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.637648 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.637761 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260927 03:17:51.638522 19219 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38281
I20260927 03:17:51.638592 19219 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:51.638782 19219 heartbeater.cc:511] Master 127.17.52.190:38281 requested a full tablet report, sending...
I20260927 03:17:51.639159 18731 ts_manager.cc:194] Registered new tserver with Master: 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:51.639539 18731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:58259
I20260927 03:17:51.641443 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 19087
I20260927 03:17:51.641530 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:17:51.641685 17618 external_mini_cluster.cc:1428] Setting key d4d2ceb52619dc8cf6194acd9c2bf49a
I20260927 03:17:51.642585 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:17:51.645779 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.639786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.17.52.131:33867)
0927 03:17:51.639920 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.639930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.639983 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.640097 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.640098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.640276 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.640321 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.640322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.641024 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.641025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.641743 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.641746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.642206 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.642207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.642297 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.642756 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.642762 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.643120 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.644859 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.644860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.644861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.645019 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.645225 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.645225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.645226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.645272 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.645533 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.645534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.645568 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.645617 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.645717 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260927 03:17:51.645826 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.639855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:33867 (local address 127.17.52.190:38281)
0927 03:17:51.639896 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.639898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.639969 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:17:51.640036 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.640037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.640046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.640085 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.640403 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.640937 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.641819 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.642108 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.642174 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.643193 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.643194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.643195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.643200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.644707 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.645093 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.645094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.645094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.645157 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.645333 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.645334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.645334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.645469 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.645540 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.645647 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.645717 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.665498 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.659027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.17.52.190:38281)
0927 03:17:51.659092 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.659095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.659121 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:17:51.659144 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.659144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.659154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.659216 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.659818 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.660320 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.661090 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.661276 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.661339 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.663401 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.663402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.663403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.663409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.664536 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.664911 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.664912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.664913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.664963 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.665120 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.665121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.665121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.665211 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.665288 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.665379 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.665436 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:17:51.667366 18731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:51.668985 18731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:51.675202 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36207 (local address 127.0.0.1:57600)
0927 03:17:51.671697 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.671705 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.671757 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.672012 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.672013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.672016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:51.672069 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.672070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673316 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.673317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.674200 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.674203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.675001 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.675002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.675057 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.675067 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.675119 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.675155 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":59,"threads_started":1}
I20260927 03:17:51.675274 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.17.52.129:36207)
0927 03:17:51.671586 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.671590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.671799 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:17:51.671839 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.671842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.671856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.671919 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:51.672155 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673231 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.674304 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.674924 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.675003 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.675056 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.675132 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.675167 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260927 03:17:51.675859 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e (DEFAULT_TABLE table=test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.676237 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545abd0afcc84819a48ce7ddd2a91c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.676446 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33519 (local address 127.0.0.1:33594)
0927 03:17:51.671280 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.671320 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.671385 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.672175 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.672176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.672178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:51.672224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.672226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673169 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.673170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.674546 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.674547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.676174 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.676175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.676256 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.676259 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.676315 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.676377 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":109,"thread_start_us":57,"threads_started":1}
I20260927 03:17:51.676506 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.17.52.131:33519)
0927 03:17:51.671871 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.671876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.671884 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:17:51.671917 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.671918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.671934 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.672113 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:51.672343 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673085 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.674626 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.675352 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.675422 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.675474 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.676370 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.676418 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":98,"threads_started":1}
I20260927 03:17:51.677241 19152 tablet_service.cc:1467] Processing CreateTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e (DEFAULT_TABLE table=test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.677574 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545abd0afcc84819a48ce7ddd2a91c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.679479 19239 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Bootstrap starting.
I20260927 03:17:51.679687 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44509 (local address 127.0.0.1:53366)
0927 03:17:51.671403 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.671412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.671451 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.671880 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.671883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.671886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:51.671950 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.671952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673290 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.673290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.674537 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.674541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.679445 (+ 4904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.679446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.679493 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.679496 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.679584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.679624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":50,"thread_start_us":65,"threads_started":1}
I20260927 03:17:51.679740 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.671586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.17.52.130:44509)
0927 03:17:51.671692 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.671695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.671703 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:17:51.671733 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.671734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.671772 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.671852 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:51.672042 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.673198 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.675508 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.676004 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.676087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.676143 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.679604 (+ 3461us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.679645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260927 03:17:51.680372 19240 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Bootstrap starting.
I20260927 03:17:51.680337 19011 tablet_service.cc:1467] Processing CreateTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e (DEFAULT_TABLE table=test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.680562 19239 tablet_bootstrap.cc:654] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.680758 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545abd0afcc84819a48ce7ddd2a91c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.681039 19239 log.cc:824] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:51.681281 19240 tablet_bootstrap.cc:654] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.681567 19240 log.cc:824] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:51.681840 19239 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: No bootstrap required, opened a new log
I20260927 03:17:51.681931 19239 ts_tablet_manager.cc:1397] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:17:51.682327 19240 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: No bootstrap required, opened a new log
I20260927 03:17:51.682453 19240 ts_tablet_manager.cc:1397] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:17:51.683924 19239 raft_consensus.cc:348] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.684089 19239 raft_consensus.cc:374] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.684124 19239 raft_consensus.cc:729] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409b9af6c9c14384b10e8c73de5ac76f, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.684242 19239 consensus_queue.cc:260] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.684496 19239 ts_tablet_manager.cc:1428] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:17:51.684434 19240 raft_consensus.cc:348] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.684525 18936 heartbeater.cc:503] Master 127.17.52.190:38281 was elected leader, sending a full tablet report...
I20260927 03:17:51.684604 19240 raft_consensus.cc:374] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.684652 19240 raft_consensus.cc:729] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ece68ca19b41d987f1846af4ee8ba0, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.684691 19243 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Bootstrap starting.
I20260927 03:17:51.684764 19240 consensus_queue.cc:260] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.685024 19240 ts_tablet_manager.cc:1428] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:17:51.685024 19219 heartbeater.cc:503] Master 127.17.52.190:38281 was elected leader, sending a full tablet report...
I20260927 03:17:51.685631 19243 tablet_bootstrap.cc:654] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.685957 19243 log.cc:824] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:51.686569 19243 tablet_bootstrap.cc:492] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: No bootstrap required, opened a new log
I20260927 03:17:51.686645 19243 ts_tablet_manager.cc:1397] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:17:51.688087 19243 raft_consensus.cc:348] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.688221 19243 raft_consensus.cc:374] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.688259 19243 raft_consensus.cc:729] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcce5472c6ca4971a49c02312c218161, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.688367 19243 consensus_queue.cc:260] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.688637 19243 ts_tablet_manager.cc:1428] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:17:51.688665 19077 heartbeater.cc:503] Master 127.17.52.190:38281 was elected leader, sending a full tablet report...
W20260927 03:17:51.726656 19078 tablet.cc:2369] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:51.833320 19244 raft_consensus.cc:482] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:51.833473 19244 raft_consensus.cc:504] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.833895 19244 leader_election.cc:288] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
W20260927 03:17:51.834633 18937 tablet.cc:2369] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:51.837625 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.834083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33519 (local address 127.17.52.129:46791)
0927 03:17:51.834123 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.834134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.834178 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.834456 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.834457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.834459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:51.834529 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.834532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.835340 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.835341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.836382 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.836385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.837373 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.837374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.837455 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.837459 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.837516 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.837554 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:17:51.837679 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.834246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:46791 (local address 127.17.52.131:33519)
0927 03:17:51.834295 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.834297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.834308 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:17:51.834340 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.834341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.834352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.834425 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:51.834613 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.835212 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.836486 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.837289 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.837374 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.837385 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.837533 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.837583 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:17:51.838156 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.834244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47557 (local address 127.17.52.130:44509)
0927 03:17:51.834289 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.834292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.834409 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:17:51.834518 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.834519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.834530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.834583 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:51.834808 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.835652 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.837186 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.837695 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.837776 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.837786 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.838060 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.838110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:17:51.838145 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545abd0afcc84819a48ce7ddd2a91c1e" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0" is_pre_election: true
I20260927 03:17:51.838310 19172 raft_consensus.cc:2454] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 0.
I20260927 03:17:51.838464 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.834084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44509 (local address 127.17.52.129:47557)
0927 03:17:51.834337 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.834345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.834421 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.834638 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.834639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.834642 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:51.834723 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.834725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.835739 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.835739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.837071 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.837073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.837835 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.837836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.837964 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.837968 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.838301 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.838376 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":142,"thread_start_us":79,"threads_started":1}
I20260927 03:17:51.838573 18821 leader_election.cc:302] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, 61ece68ca19b41d987f1846af4ee8ba0; no voters:
I20260927 03:17:51.838718 19244 raft_consensus.cc:2781] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:51.838763 19244 raft_consensus.cc:482] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:51.838792 19244 raft_consensus.cc:3037] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.838755 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545abd0afcc84819a48ce7ddd2a91c1e" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcce5472c6ca4971a49c02312c218161" is_pre_election: true
I20260927 03:17:51.838857 19031 raft_consensus.cc:2454] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 0.
I20260927 03:17:51.839847 19244 raft_consensus.cc:504] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.840011 19244 leader_election.cc:288] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 election: Requested vote from peers fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:51.840185 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545abd0afcc84819a48ce7ddd2a91c1e" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0"
I20260927 03:17:51.840261 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545abd0afcc84819a48ce7ddd2a91c1e" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcce5472c6ca4971a49c02312c218161"
I20260927 03:17:51.840281 19172 raft_consensus.cc:3037] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.840325 19031 raft_consensus.cc:3037] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.841180 19172 raft_consensus.cc:2454] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 1.
I20260927 03:17:51.841180 19031 raft_consensus.cc:2454] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 1.
I20260927 03:17:51.841351 18823 leader_election.cc:302] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, fcce5472c6ca4971a49c02312c218161; no voters:
I20260927 03:17:51.841434 19244 raft_consensus.cc:2781] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:51.841598 19244 raft_consensus.cc:686] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 LEADER]: Becoming Leader. State: Replica: 409b9af6c9c14384b10e8c73de5ac76f, State: Running, Role: LEADER
I20260927 03:17:51.841717 19244 consensus_queue.cc:237] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } }
I20260927 03:17:51.842569 18732 catalog_manager.cc:5495] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129). New cstate: current_term: 1 leader_uuid: "409b9af6c9c14384b10e8c73de5ac76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } health_report { overall_health: HEALTHY } } }
I20260927 03:17:51.856279 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.853986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.17.52.129:36207)
0927 03:17:51.854039 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.854040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.854048 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:17:51.854139 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.854140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.854154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.854209 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:51.854338 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.854785 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.855497 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.855672 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.855725 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.855935 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:51.856033 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:51.856091 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.856161 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.856178 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:17:51.857939 19031 raft_consensus.cc:1261] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Refusing update from remote peer 409b9af6c9c14384b10e8c73de5ac76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:51.857983 19172 raft_consensus.cc:1261] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Refusing update from remote peer 409b9af6c9c14384b10e8c73de5ac76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:51.858269 19249 consensus_queue.cc:1035] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:51.858395 19244 consensus_queue.cc:1035] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:51.861846 19253 mvcc.cc:205] Tried to move back new op lower bound from 7333802278327193600 to 7333802278264053760. Current Snapshot: MvccSnapshot[applied={T|T < 7333802278327193600}]
I20260927 03:17:51.863950 18731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:51.864046 18731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:51.864807 18731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Sending DeleteTablet for 3 replicas of tablet 545abd0afcc84819a48ce7ddd2a91c1e
I20260927 03:17:51.865159 19152 tablet_service.cc:1514] Processing DeleteTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:51 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:33594
I20260927 03:17:51.865381 19011 tablet_service.cc:1514] Processing DeleteTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:51 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53366
I20260927 03:17:51.865440 19262 tablet_replica.cc:330] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: stopping tablet replica
I20260927 03:17:51.865458 18731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:51.865581 18731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:51.865573 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet 545abd0afcc84819a48ce7ddd2a91c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:51 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:57600
I20260927 03:17:51.865823 19262 raft_consensus.cc:2229] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:51.865963 19263 tablet_replica.cc:330] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: stopping tablet replica
I20260927 03:17:51.865993 19262 raft_consensus.cc:2258] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:51.866050 19263 raft_consensus.cc:2229] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:51.866140 19263 raft_consensus.cc:2258] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:51.866293 19264 tablet_replica.cc:330] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: stopping tablet replica
I20260927 03:17:51.866377 19264 raft_consensus.cc:2229] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:51.866472 19264 raft_consensus.cc:2258] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:51.866714 19262 ts_tablet_manager.cc:1905] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:51.866677 18870 tablet_service.cc:1467] Processing CreateTablet for tablet e203412b5ea3490bbae5f16b419bb921 (DEFAULT_TABLE table=test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.866860 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e203412b5ea3490bbae5f16b419bb921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.866945 19264 ts_tablet_manager.cc:1905] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:51.868182 19239 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Bootstrap starting.
I20260927 03:17:51.868603 19264 ts_tablet_manager.cc:1918] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:17:51.868672 19264 log.cc:1196] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/545abd0afcc84819a48ce7ddd2a91c1e
I20260927 03:17:51.868717 19011 tablet_service.cc:1467] Processing CreateTablet for tablet e203412b5ea3490bbae5f16b419bb921 (DEFAULT_TABLE table=test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.868844 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e203412b5ea3490bbae5f16b419bb921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.869176 19262 ts_tablet_manager.cc:1918] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:17:51.869252 19263 ts_tablet_manager.cc:1905] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:51.869283 19262 log.cc:1196] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/545abd0afcc84819a48ce7ddd2a91c1e
I20260927 03:17:51.869714 19262 ts_tablet_manager.cc:1939] T 545abd0afcc84819a48ce7ddd2a91c1e P 61ece68ca19b41d987f1846af4ee8ba0: Deleting consensus metadata
I20260927 03:17:51.870360 19243 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Bootstrap starting.
I20260927 03:17:51.870667 18715 catalog_manager.cc:4841] TS 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519): tablet 545abd0afcc84819a48ce7ddd2a91c1e (table test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]) successfully deleted
I20260927 03:17:51.870746 19243 tablet_bootstrap.cc:654] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.870936 19152 tablet_service.cc:1467] Processing CreateTablet for tablet e203412b5ea3490bbae5f16b419bb921 (DEFAULT_TABLE table=test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:51.871035 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e203412b5ea3490bbae5f16b419bb921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.871148 19263 ts_tablet_manager.cc:1918] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:17:51.871219 19263 log.cc:1196] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/545abd0afcc84819a48ce7ddd2a91c1e
I20260927 03:17:51.871469 19263 ts_tablet_manager.cc:1939] T 545abd0afcc84819a48ce7ddd2a91c1e P fcce5472c6ca4971a49c02312c218161: Deleting consensus metadata
I20260927 03:17:51.871488 19243 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: No bootstrap required, opened a new log
I20260927 03:17:51.871521 19243 ts_tablet_manager.cc:1397] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:51.871667 19243 raft_consensus.cc:348] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.871728 19243 raft_consensus.cc:374] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.871747 19243 raft_consensus.cc:729] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcce5472c6ca4971a49c02312c218161, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.871790 19243 consensus_queue.cc:260] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.871856 18717 catalog_manager.cc:4841] TS fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509): tablet 545abd0afcc84819a48ce7ddd2a91c1e (table test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]) successfully deleted
I20260927 03:17:51.871883 19243 ts_tablet_manager.cc:1428] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.872205 19240 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Bootstrap starting.
I20260927 03:17:51.872332 19239 tablet_bootstrap.cc:654] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.872659 19240 tablet_bootstrap.cc:654] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.873016 19264 ts_tablet_manager.cc:1939] T 545abd0afcc84819a48ce7ddd2a91c1e P 409b9af6c9c14384b10e8c73de5ac76f: Deleting consensus metadata
I20260927 03:17:51.873286 19240 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: No bootstrap required, opened a new log
I20260927 03:17:51.873333 19240 ts_tablet_manager.cc:1397] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:51.873404 18717 catalog_manager.cc:4841] TS 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129:36207): tablet 545abd0afcc84819a48ce7ddd2a91c1e (table test-table [id=4a14da9a3bc64ae89ecb56a4e2ec5015]) successfully deleted
I20260927 03:17:51.873479 19240 raft_consensus.cc:348] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.873538 19240 raft_consensus.cc:374] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.873560 19240 raft_consensus.cc:729] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ece68ca19b41d987f1846af4ee8ba0, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.873606 19240 consensus_queue.cc:260] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.873708 19240 ts_tablet_manager.cc:1428] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.876986 19239 tablet_bootstrap.cc:492] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: No bootstrap required, opened a new log
I20260927 03:17:51.877041 19239 ts_tablet_manager.cc:1397] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260927 03:17:51.877169 19239 raft_consensus.cc:348] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.877247 19239 raft_consensus.cc:374] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.877269 19239 raft_consensus.cc:729] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409b9af6c9c14384b10e8c73de5ac76f, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.877308 19239 consensus_queue.cc:260] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.877393 19239 ts_tablet_manager.cc:1428] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260927 03:17:51.886483 19220 tablet.cc:2369] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:51.942605 19259 raft_consensus.cc:482] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:51.942740 19259 raft_consensus.cc:504] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.942950 19259 leader_election.cc:288] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:51.943126 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e203412b5ea3490bbae5f16b419bb921" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0" is_pre_election: true
I20260927 03:17:51.943132 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e203412b5ea3490bbae5f16b419bb921" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcce5472c6ca4971a49c02312c218161" is_pre_election: true
I20260927 03:17:51.943215 19172 raft_consensus.cc:2454] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 0.
I20260927 03:17:51.943248 19031 raft_consensus.cc:2454] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 0.
I20260927 03:17:51.943420 18823 leader_election.cc:302] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, fcce5472c6ca4971a49c02312c218161; no voters:
I20260927 03:17:51.943534 19259 raft_consensus.cc:2781] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:51.943567 19259 raft_consensus.cc:482] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:51.943590 19259 raft_consensus.cc:3037] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.944259 19259 raft_consensus.cc:504] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.944393 19259 leader_election.cc:288] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 election: Requested vote from peers fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:51.944536 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e203412b5ea3490bbae5f16b419bb921" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcce5472c6ca4971a49c02312c218161"
I20260927 03:17:51.944628 19031 raft_consensus.cc:3037] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.944625 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e203412b5ea3490bbae5f16b419bb921" candidate_uuid: "409b9af6c9c14384b10e8c73de5ac76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0"
I20260927 03:17:51.944707 19172 raft_consensus.cc:3037] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:51.945184 19031 raft_consensus.cc:2454] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 1.
I20260927 03:17:51.945333 18823 leader_election.cc:302] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, fcce5472c6ca4971a49c02312c218161; no voters:
I20260927 03:17:51.945407 19259 raft_consensus.cc:2781] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:51.945441 19259 raft_consensus.cc:686] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 1 LEADER]: Becoming Leader. State: Replica: 409b9af6c9c14384b10e8c73de5ac76f, State: Running, Role: LEADER
I20260927 03:17:51.945492 19259 consensus_queue.cc:237] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.945537 19172 raft_consensus.cc:2454] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409b9af6c9c14384b10e8c73de5ac76f in term 1.
I20260927 03:17:51.945960 18731 catalog_manager.cc:5495] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129). New cstate: current_term: 1 leader_uuid: "409b9af6c9c14384b10e8c73de5ac76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:17:51 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:51.987534 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.981188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38281 (local address 127.0.0.1:36812)
0927 03:17:51.981340 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:51.981349 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:51.981413 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:51.981533 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:51.981534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:51.981826 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:51.981879 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.981881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.982500 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.982500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.983019 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:51.983020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.983343 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.983344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:51.983382 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.983847 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:51.983851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:51.985130 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:51.986629 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.986635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.986636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.986819 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.986983 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:51.986983 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:51.986983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:51.987036 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:51.987246 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:51.987251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:51.987301 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:17:51.987352 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:17:51.987434 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260927 03:17:51.987545 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:51.981248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.17.52.190:38281)
0927 03:17:51.981350 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:51.981354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:51.981396 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:17:51.981466 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:51.981466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:51.981475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:51.981517 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:51.981947 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.982433 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.983088 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:51.983281 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:51.983332 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:51.985205 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:51.985206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:51.985207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:51.985213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:51.986560 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.986883 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.986883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.986884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.986929 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:51.987087 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:51.987087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:51.987088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:51.987195 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:51.987243 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:51.987383 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:17:51.987476 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":76}
I20260927 03:17:51.989631 18731 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:36812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:17:51.989800 18731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:51.991199 19152 tablet_service.cc:1467] Processing CreateTablet for tablet 01949bbd685d4caf9babea433882bd2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8055d2f390ec48eaab12af5c341020aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:51.991185 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 01949bbd685d4caf9babea433882bd2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8055d2f390ec48eaab12af5c341020aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:51.991216 19011 tablet_service.cc:1467] Processing CreateTablet for tablet 01949bbd685d4caf9babea433882bd2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8055d2f390ec48eaab12af5c341020aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:51.991346 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01949bbd685d4caf9babea433882bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.991345 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01949bbd685d4caf9babea433882bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.991356 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01949bbd685d4caf9babea433882bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:51.992403 19240 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: Bootstrap starting.
I20260927 03:17:51.992844 19243 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: Bootstrap starting.
I20260927 03:17:51.992851 19239 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: Bootstrap starting.
I20260927 03:17:51.992869 19240 tablet_bootstrap.cc:654] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.993125 19239 tablet_bootstrap.cc:654] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.993245 19243 tablet_bootstrap.cc:654] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:51.993582 19240 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: No bootstrap required, opened a new log
I20260927 03:17:51.993640 19240 ts_tablet_manager.cc:1397] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:51.993692 19239 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: No bootstrap required, opened a new log
I20260927 03:17:51.993741 19239 ts_tablet_manager.cc:1397] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:51.993861 19239 raft_consensus.cc:348] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.993852 19240 raft_consensus.cc:348] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.993918 19240 raft_consensus.cc:374] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.993918 19239 raft_consensus.cc:374] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.993940 19240 raft_consensus.cc:729] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ece68ca19b41d987f1846af4ee8ba0, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.993940 19239 raft_consensus.cc:729] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409b9af6c9c14384b10e8c73de5ac76f, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.993945 19243 tablet_bootstrap.cc:492] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: No bootstrap required, opened a new log
I20260927 03:17:51.993976 19243 ts_tablet_manager.cc:1397] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:51.993988 19239 consensus_queue.cc:260] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.993988 19240 consensus_queue.cc:260] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.994071 19239 ts_tablet_manager.cc:1428] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.994071 19240 ts_tablet_manager.cc:1428] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:51.994172 19243 raft_consensus.cc:348] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.994240 19243 raft_consensus.cc:374] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:51.994268 19243 raft_consensus.cc:729] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcce5472c6ca4971a49c02312c218161, State: Initialized, Role: FOLLOWER
I20260927 03:17:51.994314 19243 consensus_queue.cc:260] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:51.994405 19243 ts_tablet_manager.cc:1428] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:51.994457 19247 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:51.994477 19259 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:51.994519 19247 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:51.994550 19259 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:51.994663 19245 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:51.994743 19245 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:52.322872 19259 consensus_queue.cc:1035] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:52.324514 19259 consensus_queue.cc:1035] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:52.328642 19247 raft_consensus.cc:482] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:52.328720 19247 raft_consensus.cc:504] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:52.329059 19247 leader_election.cc:288] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129:36207), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:52.333050 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.329285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:51703 (local address 127.17.52.131:33519)
0927 03:17:52.329339 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.329340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.329442 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:17:52.329493 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.329494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.329504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.329554 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:52.329682 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.330436 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.332003 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.332556 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.332667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.332674 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.332933 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.332984 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:17:52.333050 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.329266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:38981 (local address 127.17.52.129:36207)
0927 03:17:52.329338 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.329341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.329375 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:17:52.329455 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.329457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.329470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.329536 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:52.329682 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.330437 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.331989 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.332551 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.332662 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.332673 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.332933 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.332981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:17:52.333042 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.329181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36207 (local address 127.17.52.130:38981)
0927 03:17:52.329296 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:52.329324 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:52.329389 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:52.329540 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:52.329541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:52.329543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:52.329619 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.329620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.330523 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.330523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.331893 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.331896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.332766 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.332767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.332834 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.332837 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:52.332890 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:17:52.332917 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":44,"thread_start_us":51,"threads_started":1}
I20260927 03:17:52.333042 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.329230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33519 (local address 127.17.52.130:51703)
0927 03:17:52.329376 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:52.329385 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:52.329435 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:52.329555 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:52.329555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:52.329556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:52.329617 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.329619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.330522 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.330523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.331893 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.331896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.332693 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.332694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.332793 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.332799 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:17:52.332866 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:17:52.332915 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":65,"thread_start_us":54,"threads_started":1}
I20260927 03:17:52.333287 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01949bbd685d4caf9babea433882bd2e" candidate_uuid: "fcce5472c6ca4971a49c02312c218161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409b9af6c9c14384b10e8c73de5ac76f" is_pre_election: true
I20260927 03:17:52.333401 18890 raft_consensus.cc:2454] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcce5472c6ca4971a49c02312c218161 in term 0.
I20260927 03:17:52.333402 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01949bbd685d4caf9babea433882bd2e" candidate_uuid: "fcce5472c6ca4971a49c02312c218161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0" is_pre_election: true
I20260927 03:17:52.333494 19172 raft_consensus.cc:2454] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcce5472c6ca4971a49c02312c218161 in term 0.
I20260927 03:17:52.333676 18964 leader_election.cc:302] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, fcce5472c6ca4971a49c02312c218161; no voters:
I20260927 03:17:52.333847 19247 raft_consensus.cc:2781] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:52.333891 19247 raft_consensus.cc:482] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:52.333910 19247 raft_consensus.cc:3037] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:52.334893 19247 raft_consensus.cc:504] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:52.335037 19247 leader_election.cc:288] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [CANDIDATE]: Term 1 election: Requested vote from peers 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129:36207), 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519)
I20260927 03:17:52.335336 19172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01949bbd685d4caf9babea433882bd2e" candidate_uuid: "fcce5472c6ca4971a49c02312c218161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ece68ca19b41d987f1846af4ee8ba0"
I20260927 03:17:52.335322 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01949bbd685d4caf9babea433882bd2e" candidate_uuid: "fcce5472c6ca4971a49c02312c218161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409b9af6c9c14384b10e8c73de5ac76f"
I20260927 03:17:52.335409 19172 raft_consensus.cc:3037] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:52.335412 18890 raft_consensus.cc:3037] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:52.336035 18890 raft_consensus.cc:2454] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcce5472c6ca4971a49c02312c218161 in term 1.
I20260927 03:17:52.336169 19172 raft_consensus.cc:2454] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcce5472c6ca4971a49c02312c218161 in term 1.
I20260927 03:17:52.336208 18964 leader_election.cc:302] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409b9af6c9c14384b10e8c73de5ac76f, fcce5472c6ca4971a49c02312c218161; no voters:
I20260927 03:17:52.336344 19247 raft_consensus.cc:2781] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:52.336479 19247 raft_consensus.cc:686] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [term 1 LEADER]: Becoming Leader. State: Replica: fcce5472c6ca4971a49c02312c218161, State: Running, Role: LEADER
I20260927 03:17:52.336536 19247 consensus_queue.cc:237] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } }
I20260927 03:17:52.336931 19282 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: TxnStatusTablet state changed. Reason: New leader fcce5472c6ca4971a49c02312c218161. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.337013 19282 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:52.337337 18731 catalog_manager.cc:5495] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcce5472c6ca4971a49c02312c218161 (127.17.52.130). New cstate: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } health_report { overall_health: UNKNOWN } } }
I20260927 03:17:52.337602 19284 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:17:52.337709 19284 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:17:52.348374 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.345715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44509 (local address 127.0.0.1:33356)
0927 03:17:52.345797 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:52.345809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:52.345861 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:52.346085 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:52.346086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:52.346087 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:52.346135 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.346137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.346829 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.346830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.347416 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.347418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.347816 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.347817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.347882 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.347907 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.348189 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.348198 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:17:52.348253 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:17:52.348305 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:17:52.348384 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.345783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.17.52.130:44509)
0927 03:17:52.345914 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.345916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.345925 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:17:52.345963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.345964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.345981 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.346064 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:52.346203 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.346741 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.347516 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.347742 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.347818 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.348010 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.348121 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.348203 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.348266 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.348290 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 03:17:52.349622 19172 raft_consensus.cc:1261] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Refusing update from remote peer fcce5472c6ca4971a49c02312c218161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:52.349725 18890 raft_consensus.cc:1261] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Refusing update from remote peer fcce5472c6ca4971a49c02312c218161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:52.350030 19282 consensus_queue.cc:1035] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:52.350170 19247 consensus_queue.cc:1035] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:52.350654 19245 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: TxnStatusTablet state changed. Reason: New leader fcce5472c6ca4971a49c02312c218161. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.350745 19245 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:52.350826 19259 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: TxnStatusTablet state changed. Reason: New leader fcce5472c6ca4971a49c02312c218161. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.350908 19259 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:52.351610 19282 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.351693 19282 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:52.351779 19282 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.351826 19282 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P fcce5472c6ca4971a49c02312c218161: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:52.351950 19252 mvcc.cc:205] Tried to move back new op lower bound from 7333802280341716992 to 7333802280291127296. Current Snapshot: MvccSnapshot[applied={T|T < 7333802280341716992}]
I20260927 03:17:52.352037 19259 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.352097 19259 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 409b9af6c9c14384b10e8c73de5ac76f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:52.352499 19254 mvcc.cc:205] Tried to move back new op lower bound from 7333802280341716992 to 7333802280291127296. Current Snapshot: MvccSnapshot[applied={T|T < 7333802280341716992}]
I20260927 03:17:52.352895 19245 tablet_replica.cc:439] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcce5472c6ca4971a49c02312c218161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409b9af6c9c14384b10e8c73de5ac76f" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36207 } } peers { permanent_uuid: "fcce5472c6ca4971a49c02312c218161" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44509 } } peers { permanent_uuid: "61ece68ca19b41d987f1846af4ee8ba0" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33519 } } }
I20260927 03:17:52.352980 19245 tablet_replica.cc:442] T 01949bbd685d4caf9babea433882bd2e P 61ece68ca19b41d987f1846af4ee8ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:52.358969 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.356065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:52181 (local address 127.17.52.130:44509)
0927 03:17:52.356116 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.356118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.356331 (+ 213us) server_negotiation.cc:407] Connection header received
0927 03:17:52.356357 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.356358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.356371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.356419 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:52.356558 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.357167 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.358300 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.358496 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.358544 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.358725 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.358753 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.358794 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.358881 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.358905 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:17:52.358996 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.355982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44509 (local address 127.17.52.129:52181)
0927 03:17:52.356198 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:52.356210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:52.356273 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:52.356435 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:52.356436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:52.356438 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:52.356494 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.356496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.357255 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.357256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.358189 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.358192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.358569 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.358570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.358629 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.358644 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.358818 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.358820 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:17:52.358868 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:17:52.358915 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
I20260927 03:17:52.369009 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.365569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36207 (local address 127.17.52.130:33803)
0927 03:17:52.365697 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:52.365708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:52.365785 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:52.365911 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:52.365912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:52.365913 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:52.365977 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.365978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.366971 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.366972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.367999 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:52.368002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.368333 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.368334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:52.368402 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.368416 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.368795 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.368797 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:17:52.368871 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:17:52.368914 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260927 03:17:52.368983 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.365656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:33803 (local address 127.17.52.129:36207)
0927 03:17:52.365700 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.365702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.365758 (+ 56us) server_negotiation.cc:407] Connection header received
0927 03:17:52.365856 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.365857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.365866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.365911 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:52.366034 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.366800 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.368073 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.368252 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.368310 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.368633 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:52.368680 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:52.368737 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.368888 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.368909 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:17:52.382103 18731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:52.382228 18731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:52.382817 18731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6da0e55de8c146c3add652d407837bef: Sending DeleteTablet for 3 replicas of tablet e203412b5ea3490bbae5f16b419bb921
I20260927 03:17:52.383068 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet e203412b5ea3490bbae5f16b419bb921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:52 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:57600
I20260927 03:17:52.383103 19011 tablet_service.cc:1514] Processing DeleteTablet for tablet e203412b5ea3490bbae5f16b419bb921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:52 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53366
I20260927 03:17:52.383260 19298 tablet_replica.cc:330] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: stopping tablet replica
I20260927 03:17:52.383289 19152 tablet_service.cc:1514] Processing DeleteTablet for tablet e203412b5ea3490bbae5f16b419bb921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:52 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:33594
I20260927 03:17:52.383325 19298 raft_consensus.cc:2229] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:52.383360 19298 raft_consensus.cc:2258] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:52.383632 19298 ts_tablet_manager.cc:1905] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:52.383915 19299 tablet_replica.cc:330] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: stopping tablet replica
I20260927 03:17:52.383981 19299 raft_consensus.cc:2229] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:52.384025 19299 raft_consensus.cc:2258] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:52.384272 19299 ts_tablet_manager.cc:1905] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:52.385295 19298 ts_tablet_manager.cc:1918] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:52.385355 19298 log.cc:1196] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/e203412b5ea3490bbae5f16b419bb921
I20260927 03:17:52.385592 19298 ts_tablet_manager.cc:1939] T e203412b5ea3490bbae5f16b419bb921 P fcce5472c6ca4971a49c02312c218161: Deleting consensus metadata
I20260927 03:17:52.385856 18717 catalog_manager.cc:4841] TS fcce5472c6ca4971a49c02312c218161 (127.17.52.130:44509): tablet e203412b5ea3490bbae5f16b419bb921 (table test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]) successfully deleted
I20260927 03:17:52.385960 19297 tablet_replica.cc:330] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: stopping tablet replica
I20260927 03:17:52.386008 19297 raft_consensus.cc:2229] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:52.386008 19299 ts_tablet_manager.cc:1918] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:52.386054 19299 log.cc:1196] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/e203412b5ea3490bbae5f16b419bb921
I20260927 03:17:52.386056 19297 raft_consensus.cc:2258] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:52.386279 19299 ts_tablet_manager.cc:1939] T e203412b5ea3490bbae5f16b419bb921 P 61ece68ca19b41d987f1846af4ee8ba0: Deleting consensus metadata
I20260927 03:17:52.386296 19297 ts_tablet_manager.cc:1905] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:52.386556 18715 catalog_manager.cc:4841] TS 61ece68ca19b41d987f1846af4ee8ba0 (127.17.52.131:33519): tablet e203412b5ea3490bbae5f16b419bb921 (table test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]) successfully deleted
I20260927 03:17:52.387857 19297 ts_tablet_manager.cc:1918] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:52.387920 19297 log.cc:1196] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/e203412b5ea3490bbae5f16b419bb921
I20260927 03:17:52.388151 19297 ts_tablet_manager.cc:1939] T e203412b5ea3490bbae5f16b419bb921 P 409b9af6c9c14384b10e8c73de5ac76f: Deleting consensus metadata
I20260927 03:17:52.388428 18717 catalog_manager.cc:4841] TS 409b9af6c9c14384b10e8c73de5ac76f (127.17.52.129:36207): tablet e203412b5ea3490bbae5f16b419bb921 (table test-table [id=9e00ebdedb974e1fb12fa09c7e50658d]) successfully deleted
Sep 27 03:17:52 dist-test-slave-7j43 krb5kdc[18685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:17:52.392776 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.384076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.17.52.129:36207)
0927 03:17:52.384114 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.384116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.384423 (+ 307us) server_negotiation.cc:407] Connection header received
0927 03:17:52.384522 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.384523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.384536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.384564 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:52.385340 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.386054 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.387226 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.387428 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.387501 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.390007 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:52.390015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:52.390015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:52.390022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:52.391588 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:52.391961 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:52.391966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:52.391967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:52.392024 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:52.392221 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:52.392222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:52.392222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:52.392362 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:52.392514 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.392596 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.392673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:17:52.392954 18930 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41772
I20260927 03:17:52.399999 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:52.393858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.17.52.190:38281)
0927 03:17:52.393899 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:52.393901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:52.393936 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:17:52.394007 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:52.394008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:52.394016 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:52.394059 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:52.394647 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.395130 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.396106 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:52.396343 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:52.396390 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:52.397628 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:52.397629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:52.397630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:52.397635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:52.399003 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:52.399377 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:52.399377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:52.399378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:52.399424 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:52.399606 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:52.399606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:52.399607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:52.399694 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:52.399772 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:52.399891 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:17:52.399942 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:17:52.400224 18731 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842
I20260927 03:17:52.400753 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18804
I20260927 03:17:52.408970 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18945
I20260927 03:17:52.414361 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 19087
I20260927 03:17:52.418759 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 18698
2026-09-27T03:17:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2542 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:52 dist-test-slave-7j43 krb5kdc[19306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:52 dist-test-slave-7j43 krb5kdc[19306](info): set up 2 sockets
Sep 27 03:17:52 dist-test-slave-7j43 krb5kdc[19306](info): commencing operation
krb5kdc: starting...
W20260927 03:17:53.460721 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:17:53.569121 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44571
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:53.650910 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:53.651093 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:53.651113 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:53.652226 19319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:17:53.652274 19319 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:17:53.652287 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:53.652333 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:53.652350 19319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:17:53.652375 19319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:17:53.653844 19319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:53.654021 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:53.654253 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:53.658859 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.658931 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.658866 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:53.659119 19319 server_base.cc:1034] running on GCE node
I20260927 03:17:53.659350 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:53.659627 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:53.660759 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790479073660768 us; error 48 us; skew 500 ppm
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:53.663041 19319 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:17:53.663393 19319 webserver.cc:460] Webserver started at http://127.17.52.190:34937/ using document root <none> and password file <none>
I20260927 03:17:53.663560 19319 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:53.663600 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:53.663683 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:53.664681 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "1647adbb005eae87eef071c873146ebe"
server_key_iv: "fba80bcbab9db0e37edc2073c5ff27ae"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.664986 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "1647adbb005eae87eef071c873146ebe"
server_key_iv: "fba80bcbab9db0e37edc2073c5ff27ae"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.666188 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:53.666906 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:53.667110 19319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:53.667207 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "1647adbb005eae87eef071c873146ebe"
server_key_iv: "fba80bcbab9db0e37edc2073c5ff27ae"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.667276 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:53.677788 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:53.679551 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:53.679684 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:53.684311 19319 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:44571
I20260927 03:17:53.684391 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:44571 every 8 connection(s)
I20260927 03:17:53.684715 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:17:53.685400 19398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:53.687642 19398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Bootstrap starting.
I20260927 03:17:53.688153 19398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:53.688377 19398 log.cc:824] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:53.689044 19398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: No bootstrap required, opened a new log
I20260927 03:17:53.691073 19398 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } }
I20260927 03:17:53.691201 19398 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:53.691222 19398 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6648cb681b4d96a4e0b7ba2c1b71ad, State: Initialized, Role: FOLLOWER
I20260927 03:17:53.691320 19398 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } }
I20260927 03:17:53.691424 19398 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:17:53.691461 19398 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:17:53.691525 19398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:53.692075 19398 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } }
I20260927 03:17:53.692193 19398 leader_election.cc:302] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad6648cb681b4d96a4e0b7ba2c1b71ad; no voters:
I20260927 03:17:53.692373 19398 leader_election.cc:288] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:17:53.692409 19401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:53.692595 19401 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [term 1 LEADER]: Becoming Leader. State: Replica: ad6648cb681b4d96a4e0b7ba2c1b71ad, State: Running, Role: LEADER
I20260927 03:17:53.692679 19398 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:17:53.692692 19401 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } }
I20260927 03:17:53.693081 19402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } } }
I20260927 03:17:53.693177 19402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:53.693292 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 19319
I20260927 03:17:53.693379 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:17:53.693560 17618 external_mini_cluster.cc:1428] Setting key 3c6d87912a7484adc4da5be2593e4494
I20260927 03:17:53.693099 19403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad6648cb681b4d96a4e0b7ba2c1b71ad. Latest consensus state: current_term: 1 leader_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6648cb681b4d96a4e0b7ba2c1b71ad" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44571 } } }
I20260927 03:17:53.693811 19403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad [sys.catalog]: This master's current role is: LEADER
W20260927 03:17:53.694572 19417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:17:53.694615 19417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:17:53.694700 19409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:17:53.694875 19409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:17:53.696246 19409 catalog_manager.cc:1347] Generated new cluster ID: 7ef42566665d4b1ba1e00baeea5368c8
I20260927 03:17:53.696303 19409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:53.705236 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:53.694023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.17.52.190:44571)
0927 03:17:53.694746 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:53.694756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:53.694789 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:17:53.694910 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:53.694918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:53.694975 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:53.695116 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:53.695862 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.697257 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.698290 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.698939 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.699048 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:53.701768 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:53.701772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:53.701772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:53.701784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:53.703587 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.703972 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.703975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.703975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.704084 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.704310 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.704311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.704311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.704642 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:53.704791 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:53.704904 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 03:17:53.705034 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":73,"threads_started":1}
I20260927 03:17:53.717883 19409 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:17:53.718624 19409 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:17:53.733237 19409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Generated new TSK 0
I20260927 03:17:53.733424 19409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:17:53.759382 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44571
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:53.838351 19424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:53.838577 19424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:53.838615 19424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:53.839715 19424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:53.839789 19424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:53.839824 19424 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:17:53.839854 19424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:53.839884 19424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:17:53.841271 19424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:53.841521 19424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:53.841732 19424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:53.846453 19432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.846530 19430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.846468 19429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:53.846637 19424 server_base.cc:1034] running on GCE node
I20260927 03:17:53.846828 19424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:53.847052 19424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:53.848191 19424 hybrid_clock.cc:648] HybridClock initialized: now 1790479073848180 us; error 52 us; skew 500 ppm
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:53.850692 19424 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:17:53.851150 19424 webserver.cc:460] Webserver started at http://127.17.52.129:46407/ using document root <none> and password file <none>
I20260927 03:17:53.851377 19424 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:53.851444 19424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:53.851601 19424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:53.852662 19424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "687dd014247d4507985bad174000b7fe"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "b68a224f8cda17f7ca65ab211ad5e819"
server_key_iv: "d229445ba7b0be1dd9274c5e6cc88f66"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.853022 19424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "687dd014247d4507985bad174000b7fe"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "b68a224f8cda17f7ca65ab211ad5e819"
server_key_iv: "d229445ba7b0be1dd9274c5e6cc88f66"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.854287 19424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:53.854976 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:53.855144 19424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:53.855239 19424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "687dd014247d4507985bad174000b7fe"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "b68a224f8cda17f7ca65ab211ad5e819"
server_key_iv: "d229445ba7b0be1dd9274c5e6cc88f66"
server_key_version: "encryptionkey@0"
I20260927 03:17:53.855327 19424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:53.866111 19424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:53.867802 19424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:53.867959 19424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:53.868484 19424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:53.868541 19424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:53.868610 19424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:53.868643 19424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:53.876029 19424 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:40809
I20260927 03:17:53.876080 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:40809 every 8 connection(s)
I20260927 03:17:53.876381 19424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:17:53.878342 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:53.868892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:37865 (local address 127.17.52.190:44571)
0927 03:17:53.868939 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:53.868941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:53.869243 (+ 302us) server_negotiation.cc:407] Connection header received
0927 03:17:53.869959 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:53.869960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:53.870005 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:53.870069 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:53.870948 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.871462 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.873123 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.873361 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.873455 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:53.875638 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:53.875640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:53.875640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:53.875647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:53.877225 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.877585 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.877586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.877586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.877641 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.877852 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.877852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.877853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.877953 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:53.878002 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:53.878214 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 03:17:53.878281 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:17:53.878482 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:53.868876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.129:37865)
0927 03:17:53.869081 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:53.869130 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:53.869857 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:53.870169 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:53.870174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:53.870551 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:53.870855 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:53.870861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.871533 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.871535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:53.873034 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:53.873038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.873476 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.873477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:53.873590 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:53.874110 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:53.874118 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:53.875549 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:53.877299 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:53.877303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:53.877305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:53.877521 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:53.877697 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:53.877698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:53.877698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:53.877798 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:53.878006 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:53.878009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:53.878109 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 03:17:53.878186 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:17:53.878297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260927 03:17:53.878996 19556 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44571
I20260927 03:17:53.879086 19556 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:53.879256 19556 heartbeater.cc:511] Master 127.17.52.190:44571 requested a full tablet report, sending...
I20260927 03:17:53.879797 19350 ts_manager.cc:194] Registered new tserver with Master: 687dd014247d4507985bad174000b7fe (127.17.52.129:40809)
I20260927 03:17:53.880631 19350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:37865
I20260927 03:17:53.884037 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 19424
I20260927 03:17:53.884138 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:17:53.884318 17618 external_mini_cluster.cc:1428] Setting key 9ca00865a6f03ddde04f810b30ffc233
I20260927 03:17:53.885208 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:53.880062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59473 (local address 127.17.52.190:44571)
0927 03:17:53.880119 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:53.880121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:53.880245 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:17:53.880270 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:53.880271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:53.880280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:53.880343 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:53.880624 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.881279 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.881953 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.882128 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.882182 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:53.882938 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:53.882939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:53.882939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:53.882944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:53.883986 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.884381 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.884382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.884383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.884441 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:53.884676 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:53.884677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:53.884677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:53.884797 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:53.884849 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:53.885064 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:17:53.885141 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:17:53.885210 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:53.880013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.129:59473)
0927 03:17:53.880179 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:53.880187 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:53.880233 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:53.880346 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:53.880347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:53.880481 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:53.880531 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:53.880532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.881375 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.881376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:53.881881 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:53.881882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:53.882192 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:53.882193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:53.882246 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:53.882570 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:53.882574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:53.882866 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:53.884063 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:53.884063 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:53.884064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:53.884294 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:53.884528 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:53.884529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:53.884529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:53.884618 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:53.884860 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:53.884861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:53.884914 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:17:53.884983 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:17:53.885084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:17:53.907191 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44571
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:53.986153 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:53.986337 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:53.986361 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:53.987491 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:53.987530 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:53.987551 19565 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:17:53.987587 19565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:53.987622 19565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:17:53.989055 19565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:53.989279 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:53.989516 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:53.994215 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.994216 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:53.994217 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:53.994537 19565 server_base.cc:1034] running on GCE node
I20260927 03:17:53.994701 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:53.994876 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:53.996009 19565 hybrid_clock.cc:648] HybridClock initialized: now 1790479073995996 us; error 27 us; skew 500 ppm
Sep 27 03:17:53 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:53.998368 19565 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:17:53.998787 19565 webserver.cc:460] Webserver started at http://127.17.52.130:43595/ using document root <none> and password file <none>
I20260927 03:17:53.998999 19565 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:53.999042 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:53.999181 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:54.000166 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "3a2bb6dd9c6749cdbd190121c15142de"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "5084d565ae1ee503bf57a3e744e165ba"
server_key_iv: "e3cbbcde8d8c9c6ee05f1ca5ad99d54f"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.000519 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "3a2bb6dd9c6749cdbd190121c15142de"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "5084d565ae1ee503bf57a3e744e165ba"
server_key_iv: "e3cbbcde8d8c9c6ee05f1ca5ad99d54f"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.001860 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:54.002502 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.002677 19565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:54.002784 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "3a2bb6dd9c6749cdbd190121c15142de"
format_stamp: "Formatted at 2026-09-27 03:17:53 on dist-test-slave-7j43"
server_key: "5084d565ae1ee503bf57a3e744e165ba"
server_key_iv: "e3cbbcde8d8c9c6ee05f1ca5ad99d54f"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.002867 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:54.017287 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:54.018959 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:54.019116 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:54.019641 19565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:54.019698 19565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.019734 19565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:54.019768 19565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:54.026821 19565 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:39475
I20260927 03:17:54.026908 19696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:39475 every 8 connection(s)
I20260927 03:17:54.027181 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:17:54.029296 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.020014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:35819 (local address 127.17.52.190:44571)
0927 03:17:54.020085 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.020087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.020336 (+ 249us) server_negotiation.cc:407] Connection header received
0927 03:17:54.020852 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.020854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.020868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.020951 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.021587 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.022416 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.023389 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.023600 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.023686 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.026198 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.026199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.026200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.026207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.027869 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.028286 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.028287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.028288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.028348 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.028646 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.028647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.028647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.028774 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.028850 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.029133 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.029208 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:17:54.029500 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.019990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.130:35819)
0927 03:17:54.020192 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.020232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.020740 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.021000 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.021003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.021222 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:54.021496 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.021501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.022504 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.022505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.023294 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.023296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.023695 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.023696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.023819 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.024307 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:54.024315 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:54.026098 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:54.027961 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.027967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.027969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.028189 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.028414 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.028415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.028415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.028528 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.028841 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:54.028852 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:54.028984 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.029095 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.029246 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":142,"threads_started":1}
I20260927 03:17:54.029963 19697 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44571
I20260927 03:17:54.030042 19697 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:54.030223 19697 heartbeater.cc:511] Master 127.17.52.190:44571 requested a full tablet report, sending...
I20260927 03:17:54.030607 19351 ts_manager.cc:194] Registered new tserver with Master: 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475)
I20260927 03:17:54.031133 19351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:35819
I20260927 03:17:54.031590 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 19565
I20260927 03:17:54.031689 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:17:54.031863 17618 external_mini_cluster.cc:1428] Setting key 7aaeff4f8434cf29957d89cd6ecb4f90
I20260927 03:17:54.036718 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.030944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:58341 (local address 127.17.52.190:44571)
0927 03:17:54.030996 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.030998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.031055 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:17:54.031116 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.031116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.031124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.031166 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.031515 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.032113 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.033079 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.033253 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.033305 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.034199 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.034200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.034201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.034208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.035533 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.035914 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.035915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.035915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.035959 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.036169 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.036170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.036170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.036326 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.036379 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.036545 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.036653 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:17:54.036761 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.030874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.130:58341)
0927 03:17:54.031000 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.031010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.031064 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.031173 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.031173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.031373 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:54.031441 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.031443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.032203 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.032204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.033004 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.033007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.033323 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.033324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.033405 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.033750 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:54.033761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:54.034120 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:54.035610 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.035611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.035612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.035842 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.036018 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.036019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.036020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.036091 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.036392 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:54.036393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:54.036443 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.036521 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.036646 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:17:54.055624 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44571
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:54.135155 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:54.135388 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:54.135427 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:54.136554 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:54.136650 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:54.136688 19706 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:17:54.136734 19706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:17:54.136767 19706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:17:54.138217 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:54.138468 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:54.138697 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:54.143435 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:54.143447 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:54.143494 19715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:54.143574 19706 server_base.cc:1034] running on GCE node
I20260927 03:17:54.143875 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:54.144132 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:54.145277 19706 hybrid_clock.cc:648] HybridClock initialized: now 1790479074145259 us; error 26 us; skew 500 ppm
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479074, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:54.147688 19706 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:17:54.148097 19706 webserver.cc:460] Webserver started at http://127.17.52.131:36359/ using document root <none> and password file <none>
I20260927 03:17:54.148274 19706 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:54.148314 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:54.148401 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:54.149477 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
format_stamp: "Formatted at 2026-09-27 03:17:54 on dist-test-slave-7j43"
server_key: "a7d0b036220c5f85bd45a6b5a88e9fd7"
server_key_iv: "9cba408a2e19d394809db22bc14d6cc2"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.149832 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
format_stamp: "Formatted at 2026-09-27 03:17:54 on dist-test-slave-7j43"
server_key: "a7d0b036220c5f85bd45a6b5a88e9fd7"
server_key_iv: "9cba408a2e19d394809db22bc14d6cc2"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.151083 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:17:54.151765 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.151913 19706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:54.151997 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
format_stamp: "Formatted at 2026-09-27 03:17:54 on dist-test-slave-7j43"
server_key: "a7d0b036220c5f85bd45a6b5a88e9fd7"
server_key_iv: "9cba408a2e19d394809db22bc14d6cc2"
server_key_version: "encryptionkey@0"
I20260927 03:17:54.152047 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:54.178282 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:54.180469 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:54.180661 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:54.181275 19706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:54.181336 19706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.181380 19706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:54.181412 19706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.188081 19706 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:34171
I20260927 03:17:54.188151 19837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:34171 every 8 connection(s)
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479074, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:54.188503 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:17:54.190250 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 19706
I20260927 03:17:54.190333 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:17:54.190562 17618 external_mini_cluster.cc:1428] Setting key 8dfa9a1c082675af976f8c9f82a4b5fd
I20260927 03:17:54.191927 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.181798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:53765 (local address 127.17.52.190:44571)
0927 03:17:54.181843 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.181845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.182042 (+ 197us) server_negotiation.cc:407] Connection header received
0927 03:17:54.182583 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.182584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.182599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.182652 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.183519 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.184441 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.186195 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.186403 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.186455 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.188937 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.188938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.188944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.188969 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.190469 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.190947 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.190949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.190949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.191005 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.191282 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.191283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.191284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.191404 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.191481 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.191759 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.191842 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:17:54.192171 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.181762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.131:53765)
0927 03:17:54.181925 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.181960 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.182469 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.182726 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.182731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.183099 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:54.183419 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.183425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.184552 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.184553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.186090 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.186093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.186487 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.186488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.186659 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.187196 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:54.187205 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:54.188840 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:54.190564 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.190570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.190572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.190881 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.191080 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.191081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.191082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.191196 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.191470 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:54.191476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:54.191612 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.191714 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.191865 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":103,"threads_started":1}
I20260927 03:17:54.192696 19838 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44571
I20260927 03:17:54.192809 19838 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:54.193001 19838 heartbeater.cc:511] Master 127.17.52.190:44571 requested a full tablet report, sending...
I20260927 03:17:54.193539 19350 ts_manager.cc:194] Registered new tserver with Master: 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:54.194062 19350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:53765
I20260927 03:17:54.198956 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.193818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:48337 (local address 127.17.52.190:44571)
0927 03:17:54.193854 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.193856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.193978 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:17:54.194019 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.194020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.194029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.194093 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.194497 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.195104 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.195783 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.195961 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.196015 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.196853 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.196854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.196854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.196860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.198170 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.198456 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.198457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.198457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.198502 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.198641 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.198641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.198641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.198718 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.198775 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.198853 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.198902 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:17:54.198974 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.193745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.17.52.131:48337)
0927 03:17:54.193906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.193916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.193971 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.194095 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.194096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.194351 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:54.194422 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.194424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.195200 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.195201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.195696 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.195697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.196045 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.196046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.196109 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.196448 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:54.196452 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:54.196783 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:54.198238 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.198238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.198239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.198392 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.198556 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.198557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.198557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.198604 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.198760 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:54.198760 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:54.198793 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.198839 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.198907 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 03:17:54.202008 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:54.223167 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.217039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.17.52.190:44571)
0927 03:17:54.217095 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.217097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.217112 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:17:54.217134 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.217135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.217143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.217190 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.217726 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.218202 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.218901 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.219072 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.219120 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.221188 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.221190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.221190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.221196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.222295 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.222590 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.222590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.222590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.222638 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.222806 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.222806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.222807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.222911 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.222980 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.223059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.223109 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:17:54.224715 19350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:54.226264 19350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:54.232486 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34171 (local address 127.0.0.1:60224)
0927 03:17:54.228548 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.228560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.228638 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.228943 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.228946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.228948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.229033 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.229035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.230048 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.230049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.231291 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.231294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232219 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.232220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.232282 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.232286 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.232319 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.232395 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":46,"threads_started":1}
I20260927 03:17:54.232515 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.17.52.129:40809)
0927 03:17:54.228903 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.228907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.228917 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:54.228955 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.228958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.228973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.229208 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.229265 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.230416 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.231533 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232059 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.232160 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.232221 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.232373 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.232413 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260927 03:17:54.232506 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.0.0.1:53150)
0927 03:17:54.228482 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.228520 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.228639 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.229075 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.229075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.229077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.229135 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.229137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.230513 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.230514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.231447 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.231450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232162 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.232164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.232241 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.232262 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.232356 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.232401 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260927 03:17:54.232852 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:39475 (local address 127.0.0.1:56240)
0927 03:17:54.228724 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.228733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.228772 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.229074 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.229075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.229077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.229135 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.229137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.230487 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.230488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.231548 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.231550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232418 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.232419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.232485 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.232488 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.232517 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.232789 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":5,"thread_start_us":60,"threads_started":1}
I20260927 03:17:54.232985 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.17.52.131:34171)
0927 03:17:54.228792 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.228797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.228807 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:54.228846 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.228848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.228863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.228939 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.229120 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.229909 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.231372 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232117 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.232725 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.232807 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.232832 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.232877 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
I20260927 03:17:54.233258 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.228560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.17.52.130:39475)
0927 03:17:54.228903 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.228907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.228917 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:54.228956 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.228958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.228973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.229208 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.229274 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.230397 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.231621 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.232330 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.233032 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.233097 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.233121 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.233156 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":75,"threads_started":1}
I20260927 03:17:54.233501 19489 tablet_service.cc:1467] Processing CreateTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 (DEFAULT_TABLE table=test-table [id=76ab03610bdd447eb982600b2091c799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.233501 19772 tablet_service.cc:1467] Processing CreateTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 (DEFAULT_TABLE table=test-table [id=76ab03610bdd447eb982600b2091c799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.233857 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e971d1dceb6a4a17a3bd681a3bfcb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.233857 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e971d1dceb6a4a17a3bd681a3bfcb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.233825 19631 tablet_service.cc:1467] Processing CreateTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 (DEFAULT_TABLE table=test-table [id=76ab03610bdd447eb982600b2091c799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.234063 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e971d1dceb6a4a17a3bd681a3bfcb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.236830 19859 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Bootstrap starting.
I20260927 03:17:54.237447 19859 tablet_bootstrap.cc:654] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.237519 19861 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Bootstrap starting.
I20260927 03:17:54.237771 19859 log.cc:824] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:54.237789 19860 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Bootstrap starting.
I20260927 03:17:54.238263 19861 tablet_bootstrap.cc:654] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.238517 19859 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: No bootstrap required, opened a new log
I20260927 03:17:54.238593 19859 ts_tablet_manager.cc:1397] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:17:54.238606 19861 log.cc:824] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:54.238715 19860 tablet_bootstrap.cc:654] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.239007 19860 log.cc:824] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:54.239648 19861 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: No bootstrap required, opened a new log
I20260927 03:17:54.239648 19860 tablet_bootstrap.cc:492] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: No bootstrap required, opened a new log
I20260927 03:17:54.239712 19860 ts_tablet_manager.cc:1397] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:17:54.239712 19861 ts_tablet_manager.cc:1397] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:17:54.240517 19859 raft_consensus.cc:348] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.240717 19859 raft_consensus.cc:374] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.240772 19859 raft_consensus.cc:729] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.240885 19859 consensus_queue.cc:260] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.241143 19859 ts_tablet_manager.cc:1428] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:17:54.241272 19697 heartbeater.cc:503] Master 127.17.52.190:44571 was elected leader, sending a full tablet report...
I20260927 03:17:54.241640 19861 raft_consensus.cc:348] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.241636 19860 raft_consensus.cc:348] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.241796 19861 raft_consensus.cc:374] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.241796 19860 raft_consensus.cc:374] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.241832 19860 raft_consensus.cc:729] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6ae9a1c154f5994ab59a8c94e5b90, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.241832 19861 raft_consensus.cc:729] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687dd014247d4507985bad174000b7fe, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.241936 19860 consensus_queue.cc:260] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.241954 19861 consensus_queue.cc:260] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.242192 19860 ts_tablet_manager.cc:1428] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:17:54.242192 19861 ts_tablet_manager.cc:1428] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:17:54.242229 19838 heartbeater.cc:503] Master 127.17.52.190:44571 was elected leader, sending a full tablet report...
I20260927 03:17:54.242328 19556 heartbeater.cc:503] Master 127.17.52.190:44571 was elected leader, sending a full tablet report...
W20260927 03:17:54.277804 19698 tablet.cc:2369] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:54.303221 19865 raft_consensus.cc:482] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:54.303349 19865 raft_consensus.cc:504] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.303663 19865 leader_election.cc:288] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171), 687dd014247d4507985bad174000b7fe (127.17.52.129:40809)
I20260927 03:17:54.306910 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.303880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.17.52.130:42571)
0927 03:17:54.303935 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.303947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.303989 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.304159 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.304160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.304162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.304239 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.304241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.305047 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.305048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.305880 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.305883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.306620 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.306620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.306707 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.306711 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.306779 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.306827 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:17:54.306917 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.303881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34171 (local address 127.17.52.130:39763)
0927 03:17:54.304061 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.304071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.304133 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.304341 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.304342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.304344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.304427 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.304429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.305234 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.305235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.305953 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.305955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.306697 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.306698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.306768 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.306771 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.306822 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.306846 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":13,"thread_start_us":111,"threads_started":1}
I20260927 03:17:54.306979 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.303930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:42571 (local address 127.17.52.129:40809)
0927 03:17:54.303973 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.303975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.303987 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:17:54.304070 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.304071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.304082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.304138 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.304348 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.304956 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.305969 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.306539 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.306626 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.306636 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.306863 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.306910 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:17:54.306979 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.303940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:39763 (local address 127.17.52.131:34171)
0927 03:17:54.303993 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.303995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.304166 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:17:54.304194 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.304195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.304205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.304266 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.304510 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.305134 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.306079 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.306625 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.306695 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.306702 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.306863 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.306910 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:17:54.307266 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e971d1dceb6a4a17a3bd681a3bfcb079" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe" is_pre_election: true
I20260927 03:17:54.307291 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e971d1dceb6a4a17a3bd681a3bfcb079" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" is_pre_election: true
I20260927 03:17:54.307443 19792 raft_consensus.cc:2454] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:54.307449 19509 raft_consensus.cc:2454] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:54.307708 19582 leader_election.cc:302] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 3a2bb6dd9c6749cdbd190121c15142de; no voters:
I20260927 03:17:54.307861 19865 raft_consensus.cc:2781] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:54.307928 19865 raft_consensus.cc:482] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:54.307946 19865 raft_consensus.cc:3037] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.308962 19865 raft_consensus.cc:504] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.309109 19865 leader_election.cc:288] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Requested vote from peers 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171), 687dd014247d4507985bad174000b7fe (127.17.52.129:40809)
I20260927 03:17:54.309248 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e971d1dceb6a4a17a3bd681a3bfcb079" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
I20260927 03:17:54.309301 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e971d1dceb6a4a17a3bd681a3bfcb079" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe"
I20260927 03:17:54.309346 19792 raft_consensus.cc:3037] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.309401 19509 raft_consensus.cc:3037] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.310057 19509 raft_consensus.cc:2454] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:54.310118 19792 raft_consensus.cc:2454] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:54.310263 19584 leader_election.cc:302] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2bb6dd9c6749cdbd190121c15142de, 687dd014247d4507985bad174000b7fe; no voters:
I20260927 03:17:54.310365 19865 raft_consensus.cc:2781] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:54.310559 19865 raft_consensus.cc:686] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Becoming Leader. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Running, Role: LEADER
I20260927 03:17:54.310664 19865 consensus_queue.cc:237] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } }
I20260927 03:17:54.311481 19350 catalog_manager.cc:5495] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130). New cstate: current_term: 1 leader_uuid: "3a2bb6dd9c6749cdbd190121c15142de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } health_report { overall_health: HEALTHY } } }
Sep 27 03:17:54 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479073, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:54.344071 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.337322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44571 (local address 127.0.0.1:45198)
0927 03:17:54.337527 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.337537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.337627 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.337822 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.337823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.338133 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:54.338185 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.338187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.338850 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.338851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.339422 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.339424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.339756 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.339757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.339801 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.340344 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:54.340349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:54.341814 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:54.343139 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.343146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.343148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.343352 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.343518 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:54.343519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:54.343519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:54.343573 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:54.343780 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:54.343786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:54.343830 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.343886 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.343973 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260927 03:17:54.344137 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.337404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.17.52.190:44571)
0927 03:17:54.337499 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.337501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.337609 (+ 108us) server_negotiation.cc:407] Connection header received
0927 03:17:54.337686 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.337687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.337697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.337813 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:54.338275 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.338772 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.339511 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.339689 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.339759 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.341896 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:54.341897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:54.341897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:54.341904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:54.343068 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.343414 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.343415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.343416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.343462 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:54.343624 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:54.343625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:54.343625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:54.343723 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:54.343785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.343915 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.344018 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":58}
I20260927 03:17:54.345968 19350 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:45198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:17:54.346167 19350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:54.347625 19631 tablet_service.cc:1467] Processing CreateTablet for tablet c51829dce24543848a7aa37e77222a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b61d5b5489b4de9a6fa9bb6b15a1ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:54.347651 19489 tablet_service.cc:1467] Processing CreateTablet for tablet c51829dce24543848a7aa37e77222a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b61d5b5489b4de9a6fa9bb6b15a1ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:54.347810 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51829dce24543848a7aa37e77222a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.347818 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51829dce24543848a7aa37e77222a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.349090 19859 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: Bootstrap starting.
I20260927 03:17:54.349246 19772 tablet_service.cc:1467] Processing CreateTablet for tablet c51829dce24543848a7aa37e77222a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b61d5b5489b4de9a6fa9bb6b15a1ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:17:54.349423 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51829dce24543848a7aa37e77222a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.349539 19859 tablet_bootstrap.cc:654] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.350235 19859 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: No bootstrap required, opened a new log
I20260927 03:17:54.350292 19859 ts_tablet_manager.cc:1397] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:54.350430 19859 raft_consensus.cc:348] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.350503 19859 raft_consensus.cc:374] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.350523 19859 raft_consensus.cc:729] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.350602 19859 consensus_queue.cc:260] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.350701 19859 ts_tablet_manager.cc:1428] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:54.350826 19865 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.350895 19865 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.350932 19861 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: Bootstrap starting.
I20260927 03:17:54.351629 19861 tablet_bootstrap.cc:654] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.351953 19860 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: Bootstrap starting.
I20260927 03:17:54.352351 19861 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: No bootstrap required, opened a new log
I20260927 03:17:54.352394 19861 ts_tablet_manager.cc:1397] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:54.352456 19860 tablet_bootstrap.cc:654] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.352541 19861 raft_consensus.cc:348] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.352648 19861 raft_consensus.cc:374] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.352679 19861 raft_consensus.cc:729] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687dd014247d4507985bad174000b7fe, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.352726 19861 consensus_queue.cc:260] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.352828 19861 ts_tablet_manager.cc:1428] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.352905 19867 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.352988 19867 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.353293 19860 tablet_bootstrap.cc:492] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: No bootstrap required, opened a new log
I20260927 03:17:54.353344 19860 ts_tablet_manager.cc:1397] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:54.353498 19860 raft_consensus.cc:348] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.353582 19860 raft_consensus.cc:374] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.353603 19860 raft_consensus.cc:729] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6ae9a1c154f5994ab59a8c94e5b90, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.353652 19860 consensus_queue.cc:260] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.353744 19860 ts_tablet_manager.cc:1428] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.353833 19866 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.353924 19866 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:17:54.377194 19557 tablet.cc:2369] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:54.386190 19867 raft_consensus.cc:482] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:54.386274 19867 raft_consensus.cc:504] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.386678 19867 leader_election.cc:288] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:54.390606 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.386850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:57267 (local address 127.17.52.131:34171)
0927 03:17:54.386894 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.386896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.387096 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:17:54.387124 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.387125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.387133 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.387188 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.387354 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.388112 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.389667 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.390217 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.390300 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.390312 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.390496 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.390535 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:17:54.390606 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.386823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:54497 (local address 127.17.52.130:39475)
0927 03:17:54.386900 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.386902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.387003 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:17:54.387061 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.387062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.387076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.387164 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:17:54.387355 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.388097 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.389668 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.390246 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.390324 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.390333 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.390494 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.390536 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:17:54.390641 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.386778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:39475 (local address 127.17.52.129:54497)
0927 03:17:54.386911 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.386924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.386991 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.387183 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.387185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.387188 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.387279 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.387280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.388188 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.388189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.389558 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.389562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.390327 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.390329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.390409 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.390417 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.390478 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.390536 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":90,"thread_start_us":73,"threads_started":1}
I20260927 03:17:54.390856 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.386779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34171 (local address 127.17.52.129:57267)
0927 03:17:54.386938 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.386947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.386997 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.387202 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.387202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.387204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:17:54.387283 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.387284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.388187 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.388188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.389558 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.389562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.390329 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.390330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.390409 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.390417 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.390478 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.390536 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":77,"thread_start_us":66,"threads_started":1}
I20260927 03:17:54.390997 19651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51829dce24543848a7aa37e77222a4a" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2bb6dd9c6749cdbd190121c15142de" is_pre_election: true
I20260927 03:17:54.391125 19651 raft_consensus.cc:2454] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 0.
I20260927 03:17:54.391355 19442 leader_election.cc:302] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2bb6dd9c6749cdbd190121c15142de, 687dd014247d4507985bad174000b7fe; no voters:
I20260927 03:17:54.391342 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51829dce24543848a7aa37e77222a4a" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" is_pre_election: true
I20260927 03:17:54.391433 19792 raft_consensus.cc:2454] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 0.
I20260927 03:17:54.391499 19867 raft_consensus.cc:2781] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:54.391551 19867 raft_consensus.cc:482] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:54.391569 19867 raft_consensus.cc:3037] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.392347 19867 raft_consensus.cc:504] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.392488 19867 leader_election.cc:288] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 election: Requested vote from peers 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:54.392693 19651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51829dce24543848a7aa37e77222a4a" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2bb6dd9c6749cdbd190121c15142de"
I20260927 03:17:54.392786 19651 raft_consensus.cc:3037] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.392763 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51829dce24543848a7aa37e77222a4a" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
I20260927 03:17:54.392839 19792 raft_consensus.cc:3037] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.393525 19792 raft_consensus.cc:2454] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 1.
I20260927 03:17:54.393605 19651 raft_consensus.cc:2454] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 1.
I20260927 03:17:54.393714 19441 leader_election.cc:302] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 687dd014247d4507985bad174000b7fe; no voters:
I20260927 03:17:54.393837 19867 raft_consensus.cc:2781] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:54.393975 19867 raft_consensus.cc:686] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [term 1 LEADER]: Becoming Leader. State: Replica: 687dd014247d4507985bad174000b7fe, State: Running, Role: LEADER
I20260927 03:17:54.394061 19867 consensus_queue.cc:237] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.394564 19882 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: TxnStatusTablet state changed. Reason: New leader 687dd014247d4507985bad174000b7fe. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.394655 19882 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:54.394906 19884 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:17:54.394930 19350 catalog_manager.cc:5495] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 687dd014247d4507985bad174000b7fe (127.17.52.129). New cstate: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260927 03:17:54.395015 19884 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:17:54.410401 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.407972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.17.52.129:40809)
0927 03:17:54.408032 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.408034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.408043 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:17:54.408115 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.408116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.408133 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.408196 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.408330 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.408875 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.409631 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.409824 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.409895 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.410074 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.410168 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.410255 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.410311 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.410330 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:17:54.410413 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.407904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.0.0.1:53152)
0927 03:17:54.407974 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.407987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.408036 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.408201 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.408201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.408202 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:54.408254 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.408266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.408953 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.408955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.409537 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.409542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.409893 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.409894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.409962 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.409975 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.410234 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.410243 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.410306 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.410348 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:17:54.412199 19792 raft_consensus.cc:1261] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Refusing update from remote peer 687dd014247d4507985bad174000b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.412220 19651 raft_consensus.cc:1261] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Refusing update from remote peer 687dd014247d4507985bad174000b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.412498 19882 consensus_queue.cc:1035] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:54.412633 19867 consensus_queue.cc:1035] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:54.413802 19866 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: TxnStatusTablet state changed. Reason: New leader 687dd014247d4507985bad174000b7fe. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.413890 19866 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.414814 19865 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: TxnStatusTablet state changed. Reason: New leader 687dd014247d4507985bad174000b7fe. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.414899 19865 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.415403 19867 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.415482 19867 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:54.415524 19867 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.415570 19867 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 687dd014247d4507985bad174000b7fe: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:17:54.415712 19865 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.415799 19865 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 3a2bb6dd9c6749cdbd190121c15142de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.416270 19866 tablet_replica.cc:439] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } } }
I20260927 03:17:54.416348 19866 tablet_replica.cc:442] T c51829dce24543848a7aa37e77222a4a P 0ea6ae9a1c154f5994ab59a8c94e5b90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:17:54.416469 19888 mvcc.cc:205] Tried to move back new op lower bound from 7333802288789786624 to 7333802288718684160. Current Snapshot: MvccSnapshot[applied={T|T < 7333802288789786624}]
I20260927 03:17:54.416628 19890 mvcc.cc:205] Tried to move back new op lower bound from 7333802288789786624 to 7333802288718684160. Current Snapshot: MvccSnapshot[applied={T|T < 7333802288789786624}]
I20260927 03:17:54.420238 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.417635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.17.52.130:39475)
0927 03:17:54.417670 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.417671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.417679 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:17:54.417772 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.417773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.417790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.417852 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.417935 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.418651 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.419377 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.419684 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.419792 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.419944 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.420055 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.420133 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.420151 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.420172 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:17:54.424801 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.422448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:57145 (local address 127.17.52.129:40809)
0927 03:17:54.422485 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.422487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.422494 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:17:54.422514 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.422514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.422522 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.422595 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.422731 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.423153 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.424166 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.424335 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.424381 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.424581 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.424626 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.424679 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.424744 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.424758 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:17:54.424858 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.422364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.17.52.130:57145)
0927 03:17:54.422433 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.422442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.422474 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.422612 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.422613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.422614 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:54.422668 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.422669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.423223 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.423224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.424089 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.424091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.424402 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.424403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.424468 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.424481 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.424684 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.424685 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.424739 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.424785 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 03:17:54.427089 19792 raft_consensus.cc:1261] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.427138 19509 raft_consensus.cc:1261] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.427358 19869 consensus_queue.cc:1035] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:54.427484 19865 consensus_queue.cc:1035] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:54.429108 19887 mvcc.cc:205] Tried to move back new op lower bound from 7333802288851668992 to 7333802288377200640. Current Snapshot: MvccSnapshot[applied={T|T < 7333802288851668992}]
I20260927 03:17:54.436193 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.433454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:52817 (local address 127.17.52.130:39475)
0927 03:17:54.433495 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.433497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.433585 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:17:54.433671 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.433672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.433683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.433732 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.433931 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.434355 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.435469 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.435754 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.435808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.436007 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.436030 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.436081 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.436137 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.436151 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:17:54.436430 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.433373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:39475 (local address 127.17.52.129:52817)
0927 03:17:54.433522 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.433533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.433609 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.433810 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.433811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.433812 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:54.433866 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.433867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.434426 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.434427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.435364 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.435366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.435845 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.435846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.435914 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.435928 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.436087 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.436088 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.436136 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.436172 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260927 03:17:54.445904 19350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:54.446014 19350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:54.446741 19350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Sending DeleteTablet for 3 replicas of tablet e971d1dceb6a4a17a3bd681a3bfcb079
I20260927 03:17:54.447064 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:54 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:60224
I20260927 03:17:54.447139 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:54 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53150
I20260927 03:17:54.447314 19631 tablet_service.cc:1514] Processing DeleteTablet for tablet e971d1dceb6a4a17a3bd681a3bfcb079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:54 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56240
I20260927 03:17:54.447562 19350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:54.447705 19350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:54.447633 19907 tablet_replica.cc:330] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: stopping tablet replica
I20260927 03:17:54.447927 19907 raft_consensus.cc:2229] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:54.448748 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee004d8c007470c86a25264f0df4b26 (DEFAULT_TABLE table=test-table [id=00ab07180bc4404f9cbdd0368cdca965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.448873 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee004d8c007470c86a25264f0df4b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.449045 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee004d8c007470c86a25264f0df4b26 (DEFAULT_TABLE table=test-table [id=00ab07180bc4404f9cbdd0368cdca965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.449158 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee004d8c007470c86a25264f0df4b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.450223 19861 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Bootstrap starting.
I20260927 03:17:54.450569 19860 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Bootstrap starting.
I20260927 03:17:54.450642 19861 tablet_bootstrap.cc:654] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.450977 19860 tablet_bootstrap.cc:654] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.448797 19631 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee004d8c007470c86a25264f0df4b26 (DEFAULT_TABLE table=test-table [id=00ab07180bc4404f9cbdd0368cdca965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:54.451452 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee004d8c007470c86a25264f0df4b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:54.452746 19859 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Bootstrap starting.
I20260927 03:17:54.453213 19859 tablet_bootstrap.cc:654] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:54.452725 19907 raft_consensus.cc:2258] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:54.453862 19859 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: No bootstrap required, opened a new log
I20260927 03:17:54.453915 19859 ts_tablet_manager.cc:1397] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:54.454069 19859 raft_consensus.cc:348] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.454147 19859 raft_consensus.cc:374] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.454175 19859 raft_consensus.cc:729] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.454217 19859 consensus_queue.cc:260] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.454300 19859 ts_tablet_manager.cc:1428] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.454358 19907 ts_tablet_manager.cc:1905] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:54.455538 19905 tablet_replica.cc:330] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: stopping tablet replica
I20260927 03:17:54.455739 19860 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: No bootstrap required, opened a new log
I20260927 03:17:54.455801 19860 ts_tablet_manager.cc:1397] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:17:54.456030 19860 raft_consensus.cc:348] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.456128 19860 raft_consensus.cc:374] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.456161 19860 raft_consensus.cc:729] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6ae9a1c154f5994ab59a8c94e5b90, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.456213 19860 consensus_queue.cc:260] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.456321 19860 ts_tablet_manager.cc:1428] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:54.456331 19861 tablet_bootstrap.cc:492] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: No bootstrap required, opened a new log
I20260927 03:17:54.456373 19861 ts_tablet_manager.cc:1397] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 03:17:54.455744 19905 raft_consensus.cc:2229] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:54.456462 19905 raft_consensus.cc:2258] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:54.456535 19861 raft_consensus.cc:348] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.456715 19861 raft_consensus.cc:374] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:54.456750 19861 raft_consensus.cc:729] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687dd014247d4507985bad174000b7fe, State: Initialized, Role: FOLLOWER
I20260927 03:17:54.456807 19861 consensus_queue.cc:260] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.456898 19905 ts_tablet_manager.cc:1905] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:54.456907 19861 ts_tablet_manager.cc:1428] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:54.458153 19907 ts_tablet_manager.cc:1918] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:54.458484 19906 tablet_replica.cc:330] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: stopping tablet replica
I20260927 03:17:54.458633 19907 log.cc:1196] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/e971d1dceb6a4a17a3bd681a3bfcb079
I20260927 03:17:54.458703 19905 ts_tablet_manager.cc:1918] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:54.458786 19905 log.cc:1196] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/e971d1dceb6a4a17a3bd681a3bfcb079
I20260927 03:17:54.459041 19905 ts_tablet_manager.cc:1939] T e971d1dceb6a4a17a3bd681a3bfcb079 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting consensus metadata
I20260927 03:17:54.459242 19906 raft_consensus.cc:2229] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:54.459347 19906 raft_consensus.cc:2258] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:54.459474 19907 ts_tablet_manager.cc:1939] T e971d1dceb6a4a17a3bd681a3bfcb079 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting consensus metadata
I20260927 03:17:54.459482 19336 catalog_manager.cc:4841] TS 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171): tablet e971d1dceb6a4a17a3bd681a3bfcb079 (table test-table [id=76ab03610bdd447eb982600b2091c799]) successfully deleted
I20260927 03:17:54.460122 19337 catalog_manager.cc:4841] TS 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475): tablet e971d1dceb6a4a17a3bd681a3bfcb079 (table test-table [id=76ab03610bdd447eb982600b2091c799]) successfully deleted
I20260927 03:17:54.460285 19906 ts_tablet_manager.cc:1905] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:54.462146 19906 ts_tablet_manager.cc:1918] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:54.462262 19906 log.cc:1196] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/e971d1dceb6a4a17a3bd681a3bfcb079
I20260927 03:17:54.462599 19906 ts_tablet_manager.cc:1939] T e971d1dceb6a4a17a3bd681a3bfcb079 P 687dd014247d4507985bad174000b7fe: Deleting consensus metadata
I20260927 03:17:54.463032 19338 catalog_manager.cc:4841] TS 687dd014247d4507985bad174000b7fe (127.17.52.129:40809): tablet e971d1dceb6a4a17a3bd681a3bfcb079 (table test-table [id=76ab03610bdd447eb982600b2091c799]) successfully deleted
I20260927 03:17:54.530287 19882 raft_consensus.cc:482] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:54.530401 19882 raft_consensus.cc:504] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.530578 19882 leader_election.cc:288] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:54.530830 19651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee004d8c007470c86a25264f0df4b26" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2bb6dd9c6749cdbd190121c15142de" is_pre_election: true
I20260927 03:17:54.530845 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee004d8c007470c86a25264f0df4b26" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" is_pre_election: true
I20260927 03:17:54.530929 19651 raft_consensus.cc:2454] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 0.
I20260927 03:17:54.530937 19792 raft_consensus.cc:2454] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 0.
I20260927 03:17:54.531118 19441 leader_election.cc:302] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 687dd014247d4507985bad174000b7fe; no voters:
I20260927 03:17:54.531201 19882 raft_consensus.cc:2781] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:54.531229 19882 raft_consensus.cc:482] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:54.531247 19882 raft_consensus.cc:3037] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.532174 19882 raft_consensus.cc:504] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.532306 19882 leader_election.cc:288] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 election: Requested vote from peers 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:54.532442 19651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee004d8c007470c86a25264f0df4b26" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2bb6dd9c6749cdbd190121c15142de"
I20260927 03:17:54.532505 19651 raft_consensus.cc:3037] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.532480 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee004d8c007470c86a25264f0df4b26" candidate_uuid: "687dd014247d4507985bad174000b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
I20260927 03:17:54.532542 19792 raft_consensus.cc:3037] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:54.533325 19651 raft_consensus.cc:2454] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 1.
I20260927 03:17:54.533355 19792 raft_consensus.cc:2454] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687dd014247d4507985bad174000b7fe in term 1.
I20260927 03:17:54.533512 19441 leader_election.cc:302] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 687dd014247d4507985bad174000b7fe; no voters:
I20260927 03:17:54.533601 19882 raft_consensus.cc:2781] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:54.533638 19882 raft_consensus.cc:686] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 1 LEADER]: Becoming Leader. State: Replica: 687dd014247d4507985bad174000b7fe, State: Running, Role: LEADER
I20260927 03:17:54.533680 19882 consensus_queue.cc:237] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:54.534226 19350 catalog_manager.cc:5495] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 687dd014247d4507985bad174000b7fe (127.17.52.129). New cstate: current_term: 1 leader_uuid: "687dd014247d4507985bad174000b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260927 03:17:54.548010 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.545532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.17.52.129:40809)
0927 03:17:54.545604 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.545606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.545616 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:17:54.545639 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.545640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.545652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.545718 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.545846 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.546551 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.547332 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.547608 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.547683 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.547787 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.547820 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.547867 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.547914 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.547935 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:17:54.552268 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.550220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:41677 (local address 127.17.52.129:40809)
0927 03:17:54.550275 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:54.550277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:54.550284 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:17:54.550304 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:54.550305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:54.550313 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:54.550365 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:54.550455 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.550888 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.551605 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.551816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.551890 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.552017 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.552045 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.552094 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:54.552172 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:17:54.552187 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:17:54.552268 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:54.550136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.17.52.129:41677)
0927 03:17:54.550186 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:54.550197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:54.550256 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:54.550367 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:54.550368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:54.550369 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:17:54.550402 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.550403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.550971 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.550972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.551506 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:54.551508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:54.551886 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:54.551887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:54.551935 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:54.551942 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:17:54.552104 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:17:54.552105 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:17:54.552165 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:17:54.552206 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:17:54.554313 19651 raft_consensus.cc:1261] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Refusing update from remote peer 687dd014247d4507985bad174000b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.554529 19792 raft_consensus.cc:1261] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Refusing update from remote peer 687dd014247d4507985bad174000b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:54.554518 19867 consensus_queue.cc:1035] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:54.554708 19867 consensus_queue.cc:1035] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:17:54.559402 19887 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:17:56.078682 19350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:56.078815 19350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:56.079794 19350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Sending DeleteTablet for 3 replicas of tablet 5ee004d8c007470c86a25264f0df4b26
I20260927 03:17:56.080053 19631 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee004d8c007470c86a25264f0df4b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56240
I20260927 03:17:56.080250 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee004d8c007470c86a25264f0df4b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53150
I20260927 03:17:56.080406 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee004d8c007470c86a25264f0df4b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:60224
I20260927 03:17:56.080610 19938 tablet_replica.cc:330] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: stopping tablet replica
I20260927 03:17:56.080677 19938 raft_consensus.cc:2229] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:56.080718 19938 raft_consensus.cc:2258] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.080965 19938 ts_tablet_manager.cc:1905] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.082537 19938 ts_tablet_manager.cc:1918] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:17:56.082614 19938 log.cc:1196] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/5ee004d8c007470c86a25264f0df4b26
I20260927 03:17:56.082681 19937 tablet_replica.cc:330] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: stopping tablet replica
I20260927 03:17:56.082747 19937 raft_consensus.cc:2229] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:56.082854 19937 raft_consensus.cc:2258] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.082863 19938 ts_tablet_manager.cc:1939] T 5ee004d8c007470c86a25264f0df4b26 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting consensus metadata
I20260927 03:17:56.082980 19936 tablet_replica.cc:330] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: stopping tablet replica
I20260927 03:17:56.083040 19936 raft_consensus.cc:2229] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:56.083081 19936 raft_consensus.cc:2258] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.083173 19336 catalog_manager.cc:4841] TS 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171): tablet 5ee004d8c007470c86a25264f0df4b26 (table test-table [id=00ab07180bc4404f9cbdd0368cdca965]) successfully deleted
I20260927 03:17:56.083263 19937 ts_tablet_manager.cc:1905] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.084734 19937 ts_tablet_manager.cc:1918] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:17:56.084748 19936 ts_tablet_manager.cc:1905] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.084812 19937 log.cc:1196] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/5ee004d8c007470c86a25264f0df4b26
I20260927 03:17:56.085067 19937 ts_tablet_manager.cc:1939] T 5ee004d8c007470c86a25264f0df4b26 P 687dd014247d4507985bad174000b7fe: Deleting consensus metadata
I20260927 03:17:56.085361 19338 catalog_manager.cc:4841] TS 687dd014247d4507985bad174000b7fe (127.17.52.129:40809): tablet 5ee004d8c007470c86a25264f0df4b26 (table test-table [id=00ab07180bc4404f9cbdd0368cdca965]) successfully deleted
I20260927 03:17:56.086198 19936 ts_tablet_manager.cc:1918] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:17:56.086248 19936 log.cc:1196] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/5ee004d8c007470c86a25264f0df4b26
I20260927 03:17:56.086467 19936 ts_tablet_manager.cc:1939] T 5ee004d8c007470c86a25264f0df4b26 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting consensus metadata
I20260927 03:17:56.086759 19337 catalog_manager.cc:4841] TS 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475): tablet 5ee004d8c007470c86a25264f0df4b26 (table test-table [id=00ab07180bc4404f9cbdd0368cdca965]) successfully deleted
Sep 27 03:17:56 dist-test-slave-7j43 krb5kdc[19306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:17:56 dist-test-slave-7j43 krb5kdc[19306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:56.107132 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:56.100591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.17.52.190:44571)
0927 03:17:56.100738 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:56.100740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:56.100770 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:17:56.100835 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:56.100836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:56.100847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:56.100905 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:56.101432 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:56.101942 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:56.102631 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:56.102816 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:56.102863 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:56.104958 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:56.104959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:56.104959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:56.104966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:56.106272 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:56.106558 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:56.106558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:56.106559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:56.106610 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:56.106762 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:56.106762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:56.106762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:56.106850 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:56.106917 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:56.107008 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:17:56.107066 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":103,"threads_started":1}
I20260927 03:17:56.108677 19350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:56.108843 19350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:56.110157 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 953db47a4d614e91b945a3c0bcee5348 (DEFAULT_TABLE table=test-table [id=e419ae65b5f64f999d562d322345ee4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.110180 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 953db47a4d614e91b945a3c0bcee5348 (DEFAULT_TABLE table=test-table [id=e419ae65b5f64f999d562d322345ee4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.110234 19631 tablet_service.cc:1467] Processing CreateTablet for tablet 953db47a4d614e91b945a3c0bcee5348 (DEFAULT_TABLE table=test-table [id=e419ae65b5f64f999d562d322345ee4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.110307 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953db47a4d614e91b945a3c0bcee5348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.110313 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953db47a4d614e91b945a3c0bcee5348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.110339 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953db47a4d614e91b945a3c0bcee5348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.111352 19947 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Bootstrap starting.
I20260927 03:17:56.111577 19948 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Bootstrap starting.
I20260927 03:17:56.111758 19949 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Bootstrap starting.
I20260927 03:17:56.111896 19948 tablet_bootstrap.cc:654] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.111896 19947 tablet_bootstrap.cc:654] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.112135 19949 tablet_bootstrap.cc:654] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.112538 19948 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: No bootstrap required, opened a new log
I20260927 03:17:56.112538 19947 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: No bootstrap required, opened a new log
I20260927 03:17:56.112625 19947 ts_tablet_manager.cc:1397] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:56.112666 19948 ts_tablet_manager.cc:1397] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:56.112787 19949 tablet_bootstrap.cc:492] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: No bootstrap required, opened a new log
I20260927 03:17:56.112776 19947 raft_consensus.cc:348] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.112775 19948 raft_consensus.cc:348] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.112840 19948 raft_consensus.cc:374] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.112840 19947 raft_consensus.cc:374] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.112840 19949 ts_tablet_manager.cc:1397] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:56.112864 19947 raft_consensus.cc:729] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687dd014247d4507985bad174000b7fe, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.112864 19948 raft_consensus.cc:729] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.112913 19947 consensus_queue.cc:260] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.112951 19948 consensus_queue.cc:260] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.113067 19947 ts_tablet_manager.cc:1428] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:56.113122 19948 ts_tablet_manager.cc:1428] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:56.112963 19949 raft_consensus.cc:348] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.113386 19949 raft_consensus.cc:374] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.113603 19949 raft_consensus.cc:729] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6ae9a1c154f5994ab59a8c94e5b90, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.113664 19949 consensus_queue.cc:260] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.113847 19949 ts_tablet_manager.cc:1428] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 03:17:56.190313 19839 tablet.cc:2369] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:17:56.220505 19953 raft_consensus.cc:482] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:56.220619 19953 raft_consensus.cc:504] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.220801 19953 leader_election.cc:288] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 687dd014247d4507985bad174000b7fe (127.17.52.129:40809), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:56.221038 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953db47a4d614e91b945a3c0bcee5348" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe" is_pre_election: true
I20260927 03:17:56.221084 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953db47a4d614e91b945a3c0bcee5348" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" is_pre_election: true
I20260927 03:17:56.221156 19509 raft_consensus.cc:2454] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:56.221197 19792 raft_consensus.cc:2454] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:56.221360 19582 leader_election.cc:302] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 3a2bb6dd9c6749cdbd190121c15142de; no voters:
I20260927 03:17:56.221458 19953 raft_consensus.cc:2781] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:56.221496 19953 raft_consensus.cc:482] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:56.221522 19953 raft_consensus.cc:3037] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.222214 19953 raft_consensus.cc:504] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.222357 19953 leader_election.cc:288] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Requested vote from peers 687dd014247d4507985bad174000b7fe (127.17.52.129:40809), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:56.222512 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953db47a4d614e91b945a3c0bcee5348" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
I20260927 03:17:56.222579 19792 raft_consensus.cc:3037] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.222534 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953db47a4d614e91b945a3c0bcee5348" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe"
I20260927 03:17:56.222617 19509 raft_consensus.cc:3037] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.223137 19792 raft_consensus.cc:2454] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:56.223296 19582 leader_election.cc:302] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 3a2bb6dd9c6749cdbd190121c15142de; no voters:
I20260927 03:17:56.223313 19509 raft_consensus.cc:2454] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:56.223407 19953 raft_consensus.cc:2781] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:56.223567 19953 raft_consensus.cc:686] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Becoming Leader. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Running, Role: LEADER
I20260927 03:17:56.223634 19953 consensus_queue.cc:237] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.224252 19350 catalog_manager.cc:5495] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130). New cstate: current_term: 1 leader_uuid: "3a2bb6dd9c6749cdbd190121c15142de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260927 03:17:56.230906 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:56.228357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.17.52.130:39475)
0927 03:17:56.228490 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:56.228492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:56.228498 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:17:56.228517 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:56.228518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:56.228528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:56.228646 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:17:56.228759 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:56.229244 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:56.230109 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:56.230327 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:56.230409 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:56.230558 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:17:56.230597 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:17:56.230681 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:56.230836 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:17:56.230853 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260927 03:17:56.232867 19792 raft_consensus.cc:1261] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:56.232868 19509 raft_consensus.cc:1261] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:56.233054 19953 consensus_queue.cc:1035] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:56.233139 19955 consensus_queue.cc:1035] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:56.243083 19351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:56.243206 19351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:56.243901 19351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Sending DeleteTablet for 3 replicas of tablet 953db47a4d614e91b945a3c0bcee5348
I20260927 03:17:56.244134 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 953db47a4d614e91b945a3c0bcee5348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53150
I20260927 03:17:56.244205 19631 tablet_service.cc:1514] Processing DeleteTablet for tablet 953db47a4d614e91b945a3c0bcee5348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56240
I20260927 03:17:56.244235 19937 tablet_replica.cc:330] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: stopping tablet replica
I20260927 03:17:56.244242 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet 953db47a4d614e91b945a3c0bcee5348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:56 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:60224
I20260927 03:17:56.244293 19937 raft_consensus.cc:2229] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:56.244325 19937 raft_consensus.cc:2258] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.244329 19938 tablet_replica.cc:330] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: stopping tablet replica
I20260927 03:17:56.244400 19938 raft_consensus.cc:2229] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:56.244402 19936 tablet_replica.cc:330] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: stopping tablet replica
I20260927 03:17:56.244447 19938 raft_consensus.cc:2258] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.244457 19936 raft_consensus.cc:2229] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:56.244521 19936 raft_consensus.cc:2258] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:56.244831 19938 ts_tablet_manager.cc:1905] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.244894 19936 ts_tablet_manager.cc:1905] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.244897 19351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:17:56.245026 19351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:17:56.246384 19938 ts_tablet_manager.cc:1918] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:56.246434 19938 log.cc:1196] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/953db47a4d614e91b945a3c0bcee5348
I20260927 03:17:56.246441 19772 tablet_service.cc:1467] Processing CreateTablet for tablet e68b34c4f40848a69a0d9a85db174911 (DEFAULT_TABLE table=test-table [id=ddfe4c4aeede4d199f5a3d45c6f3816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.246526 19936 ts_tablet_manager.cc:1918] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:56.246547 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68b34c4f40848a69a0d9a85db174911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.246570 19936 log.cc:1196] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/953db47a4d614e91b945a3c0bcee5348
I20260927 03:17:56.246614 19938 ts_tablet_manager.cc:1939] T 953db47a4d614e91b945a3c0bcee5348 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting consensus metadata
I20260927 03:17:56.246783 19936 ts_tablet_manager.cc:1939] T 953db47a4d614e91b945a3c0bcee5348 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting consensus metadata
I20260927 03:17:56.247102 19337 catalog_manager.cc:4841] TS 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475): tablet 953db47a4d614e91b945a3c0bcee5348 (table test-table [id=e419ae65b5f64f999d562d322345ee4a]) successfully deleted
I20260927 03:17:56.247201 19937 ts_tablet_manager.cc:1905] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:56.247750 19949 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Bootstrap starting.
I20260927 03:17:56.247884 19336 catalog_manager.cc:4841] TS 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171): tablet 953db47a4d614e91b945a3c0bcee5348 (table test-table [id=e419ae65b5f64f999d562d322345ee4a]) successfully deleted
I20260927 03:17:56.248008 19631 tablet_service.cc:1467] Processing CreateTablet for tablet e68b34c4f40848a69a0d9a85db174911 (DEFAULT_TABLE table=test-table [id=ddfe4c4aeede4d199f5a3d45c6f3816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.248070 19949 tablet_bootstrap.cc:654] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.248097 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68b34c4f40848a69a0d9a85db174911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.248759 19937 ts_tablet_manager.cc:1918] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:17:56.248822 19937 log.cc:1196] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/953db47a4d614e91b945a3c0bcee5348
I20260927 03:17:56.249033 19937 ts_tablet_manager.cc:1939] T 953db47a4d614e91b945a3c0bcee5348 P 687dd014247d4507985bad174000b7fe: Deleting consensus metadata
I20260927 03:17:56.249150 19949 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: No bootstrap required, opened a new log
I20260927 03:17:56.249209 19949 ts_tablet_manager.cc:1397] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:17:56.249286 19489 tablet_service.cc:1467] Processing CreateTablet for tablet e68b34c4f40848a69a0d9a85db174911 (DEFAULT_TABLE table=test-table [id=ddfe4c4aeede4d199f5a3d45c6f3816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:17:56.249356 19948 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Bootstrap starting.
I20260927 03:17:56.249397 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68b34c4f40848a69a0d9a85db174911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:56.249377 19949 raft_consensus.cc:348] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.249454 19949 raft_consensus.cc:374] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.249483 19949 raft_consensus.cc:729] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6ae9a1c154f5994ab59a8c94e5b90, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.249528 19949 consensus_queue.cc:260] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.249610 19949 ts_tablet_manager.cc:1428] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:56.249652 19948 tablet_bootstrap.cc:654] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.249927 19338 catalog_manager.cc:4841] TS 687dd014247d4507985bad174000b7fe (127.17.52.129:40809): tablet 953db47a4d614e91b945a3c0bcee5348 (table test-table [id=e419ae65b5f64f999d562d322345ee4a]) successfully deleted
I20260927 03:17:56.250563 19947 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe: Bootstrap starting.
I20260927 03:17:56.250823 19947 tablet_bootstrap.cc:654] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:56.250937 19948 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: No bootstrap required, opened a new log
I20260927 03:17:56.250993 19948 ts_tablet_manager.cc:1397] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:17:56.251097 19948 raft_consensus.cc:348] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.251181 19948 raft_consensus.cc:374] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.251204 19948 raft_consensus.cc:729] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.251257 19948 consensus_queue.cc:260] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.251343 19947 tablet_bootstrap.cc:492] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe: No bootstrap required, opened a new log
I20260927 03:17:56.251353 19948 ts_tablet_manager.cc:1428] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:56.251386 19947 ts_tablet_manager.cc:1397] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:17:56.251523 19947 raft_consensus.cc:348] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.251592 19947 raft_consensus.cc:374] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:56.251613 19947 raft_consensus.cc:729] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687dd014247d4507985bad174000b7fe, State: Initialized, Role: FOLLOWER
I20260927 03:17:56.251658 19947 consensus_queue.cc:260] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.251753 19947 ts_tablet_manager.cc:1428] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:56.344837 19953 raft_consensus.cc:482] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:17:56.344928 19953 raft_consensus.cc:504] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.345074 19953 leader_election.cc:288] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 687dd014247d4507985bad174000b7fe (127.17.52.129:40809), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:56.345278 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68b34c4f40848a69a0d9a85db174911" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" is_pre_election: true
I20260927 03:17:56.345304 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68b34c4f40848a69a0d9a85db174911" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe" is_pre_election: true
I20260927 03:17:56.345396 19792 raft_consensus.cc:2454] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:56.345392 19509 raft_consensus.cc:2454] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 0.
I20260927 03:17:56.345562 19582 leader_election.cc:302] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 3a2bb6dd9c6749cdbd190121c15142de; no voters:
I20260927 03:17:56.345685 19953 raft_consensus.cc:2781] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:17:56.345736 19953 raft_consensus.cc:482] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:17:56.345759 19953 raft_consensus.cc:3037] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.346395 19953 raft_consensus.cc:504] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.346530 19953 leader_election.cc:288] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Requested vote from peers 687dd014247d4507985bad174000b7fe (127.17.52.129:40809), 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171)
I20260927 03:17:56.346665 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68b34c4f40848a69a0d9a85db174911" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90"
I20260927 03:17:56.346666 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68b34c4f40848a69a0d9a85db174911" candidate_uuid: "3a2bb6dd9c6749cdbd190121c15142de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687dd014247d4507985bad174000b7fe"
I20260927 03:17:56.346735 19509 raft_consensus.cc:3037] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.346735 19792 raft_consensus.cc:3037] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:56.347435 19792 raft_consensus.cc:2454] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:56.347470 19509 raft_consensus.cc:2454] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2bb6dd9c6749cdbd190121c15142de in term 1.
I20260927 03:17:56.347615 19582 leader_election.cc:302] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6ae9a1c154f5994ab59a8c94e5b90, 3a2bb6dd9c6749cdbd190121c15142de; no voters:
I20260927 03:17:56.347827 19953 raft_consensus.cc:2781] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:56.347895 19953 raft_consensus.cc:686] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Becoming Leader. State: Replica: 3a2bb6dd9c6749cdbd190121c15142de, State: Running, Role: LEADER
I20260927 03:17:56.347949 19953 consensus_queue.cc:237] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } }
I20260927 03:17:56.348501 19350 catalog_manager.cc:5495] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130). New cstate: current_term: 1 leader_uuid: "3a2bb6dd9c6749cdbd190121c15142de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2bb6dd9c6749cdbd190121c15142de" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 39475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260927 03:17:56.370500 19509 raft_consensus.cc:1261] T e68b34c4f40848a69a0d9a85db174911 P 687dd014247d4507985bad174000b7fe [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:56.370532 19792 raft_consensus.cc:1261] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Refusing update from remote peer 3a2bb6dd9c6749cdbd190121c15142de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:17:56.370760 19953 consensus_queue.cc:1035] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6ae9a1c154f5994ab59a8c94e5b90" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:17:56.370890 19955 consensus_queue.cc:1035] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [LEADER]: Connected to new peer: Peer: permanent_uuid: "687dd014247d4507985bad174000b7fe" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:17:56.374780 19920 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:17:57.891556 19349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:57.891655 19349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:17:57.892628 19349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad6648cb681b4d96a4e0b7ba2c1b71ad: Sending DeleteTablet for 3 replicas of tablet e68b34c4f40848a69a0d9a85db174911
I20260927 03:17:57.892920 19631 tablet_service.cc:1514] Processing DeleteTablet for tablet e68b34c4f40848a69a0d9a85db174911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:57 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56240
I20260927 03:17:57.892915 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet e68b34c4f40848a69a0d9a85db174911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:57 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:53150
I20260927 03:17:57.893039 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 19424
I20260927 03:17:57.893110 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet e68b34c4f40848a69a0d9a85db174911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:17:57 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:60224
I20260927 03:17:57.894026 19994 tablet_replica.cc:330] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: stopping tablet replica
I20260927 03:17:57.894098 19994 raft_consensus.cc:2229] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:17:57.894145 19994 raft_consensus.cc:2258] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:57.894245 19992 tablet_replica.cc:330] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: stopping tablet replica
I20260927 03:17:57.894315 19992 raft_consensus.cc:2229] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:17:57.894379 19992 raft_consensus.cc:2258] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:17:57.894578 19994 ts_tablet_manager.cc:1905] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:57.896505 19994 ts_tablet_manager.cc:1918] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:17:57.896595 19994 log.cc:1196] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/e68b34c4f40848a69a0d9a85db174911
I20260927 03:17:57.896821 19994 ts_tablet_manager.cc:1939] T e68b34c4f40848a69a0d9a85db174911 P 0ea6ae9a1c154f5994ab59a8c94e5b90: Deleting consensus metadata
I20260927 03:17:57.897176 19336 catalog_manager.cc:4841] TS 0ea6ae9a1c154f5994ab59a8c94e5b90 (127.17.52.131:34171): tablet e68b34c4f40848a69a0d9a85db174911 (table test-table [id=ddfe4c4aeede4d199f5a3d45c6f3816a]) successfully deleted
I20260927 03:17:57.897202 19992 ts_tablet_manager.cc:1905] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:17:57.899055 19992 ts_tablet_manager.cc:1918] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:17:57.899117 19992 log.cc:1196] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/e68b34c4f40848a69a0d9a85db174911
I20260927 03:17:57.899361 19992 ts_tablet_manager.cc:1939] T e68b34c4f40848a69a0d9a85db174911 P 3a2bb6dd9c6749cdbd190121c15142de: Deleting consensus metadata
I20260927 03:17:57.899654 19337 catalog_manager.cc:4841] TS 3a2bb6dd9c6749cdbd190121c15142de (127.17.52.130:39475): tablet e68b34c4f40848a69a0d9a85db174911 (table test-table [id=ddfe4c4aeede4d199f5a3d45c6f3816a]) successfully deleted
W20260927 03:17:57.900524 19338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:40809 (error 108)
I20260927 03:17:57.901114 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 19565
I20260927 03:17:57.901126 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:57.900963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40809 (local address 127.0.0.1:53170)
0927 03:17:57.901061 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:57.901095 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:40809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":64,"threads_started":1}
W20260927 03:17:57.901852 19338 catalog_manager.cc:4584] TS 687dd014247d4507985bad174000b7fe (127.17.52.129:40809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e68b34c4f40848a69a0d9a85db174911: Network error: Client connection negotiation failed: client connection to 127.17.52.129:40809: connect: Connection refused (error 111)
I20260927 03:17:57.907459 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 19706
I20260927 03:17:57.911978 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 19319
2026-09-27T03:17:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5493 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:57 dist-test-slave-7j43 krb5kdc[19999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:57 dist-test-slave-7j43 krb5kdc[19999](info): set up 2 sockets
Sep 27 03:17:57 dist-test-slave-7j43 krb5kdc[19999](info): commencing operation
krb5kdc: starting...
W20260927 03:17:58.940872 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:17:59.054510 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45155
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:59.134480 20012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:59.134668 20012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:59.134690 20012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:59.135767 20012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:17:59.135802 20012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:59.135816 20012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:59.135829 20012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:17:59.135841 20012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:17:59.137210 20012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:59.137408 20012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:59.137647 20012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:59.142161 20018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.142199 20017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.142207 20020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:59.142462 20012 server_base.cc:1034] running on GCE node
I20260927 03:17:59.142679 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:59.142905 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:59.144083 20012 hybrid_clock.cc:648] HybridClock initialized: now 1790479079144065 us; error 31 us; skew 500 ppm
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:59.150521 20012 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:17:59.150954 20012 webserver.cc:460] Webserver started at http://127.17.52.190:42603/ using document root <none> and password file <none>
I20260927 03:17:59.151129 20012 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:59.151176 20012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:59.151278 20012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:59.152242 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "7988c62bd6ce4778bd01bce4f7411311"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "7729f49055be59f755becf5df75c5ed5"
server_key_iv: "00cd22733aadbec793f6f100c0c66466"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.152559 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "7988c62bd6ce4778bd01bce4f7411311"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "7729f49055be59f755becf5df75c5ed5"
server_key_iv: "00cd22733aadbec793f6f100c0c66466"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.153858 20012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:59.154507 20027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.154670 20012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:17:59.154767 20012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "7988c62bd6ce4778bd01bce4f7411311"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "7729f49055be59f755becf5df75c5ed5"
server_key_iv: "00cd22733aadbec793f6f100c0c66466"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.154840 20012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:59.197722 20012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:59.199853 20012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:59.200042 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:59.204455 20012 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:45155
I20260927 03:17:59.204505 20079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:45155 every 8 connection(s)
I20260927 03:17:59.204906 20012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:17:59.205832 20080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:17:59.208074 20080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: Bootstrap starting.
I20260927 03:17:59.208647 20080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: Neither blocks nor log segments found. Creating new log.
I20260927 03:17:59.208882 20080 log.cc:824] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: Log is configured to *not* fsync() on all Append() calls
I20260927 03:17:59.209566 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20012
I20260927 03:17:59.209666 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:17:59.209743 20080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: No bootstrap required, opened a new log
I20260927 03:17:59.209851 17618 external_mini_cluster.cc:1428] Setting key 5d03deba7f9473dd7f94e577dd7674ff
I20260927 03:17:59.212401 20080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } }
I20260927 03:17:59.212545 20080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:17:59.212615 20080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7988c62bd6ce4778bd01bce4f7411311, State: Initialized, Role: FOLLOWER
I20260927 03:17:59.212749 20080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } }
I20260927 03:17:59.212850 20080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:17:59.212872 20080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:17:59.212941 20080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:17:59.213526 20080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } }
I20260927 03:17:59.213620 20080 leader_election.cc:302] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7988c62bd6ce4778bd01bce4f7411311; no voters:
I20260927 03:17:59.213867 20080 leader_election.cc:288] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:17:59.213891 20085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:17:59.214001 20085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [term 1 LEADER]: Becoming Leader. State: Replica: 7988c62bd6ce4778bd01bce4f7411311, State: Running, Role: LEADER
I20260927 03:17:59.214133 20080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:17:59.214124 20085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } }
I20260927 03:17:59.215644 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7988c62bd6ce4778bd01bce4f7411311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } } }
I20260927 03:17:59.215736 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [sys.catalog]: This master's current role is: LEADER
I20260927 03:17:59.215952 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7988c62bd6ce4778bd01bce4f7411311. Latest consensus state: current_term: 1 leader_uuid: "7988c62bd6ce4778bd01bce4f7411311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7988c62bd6ce4778bd01bce4f7411311" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45155 } } }
I20260927 03:17:59.216006 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311 [sys.catalog]: This master's current role is: LEADER
W20260927 03:17:59.216408 20099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:17:59.216464 20099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:17:59.216527 20100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:17:59.216727 20100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:59.218303 20100 catalog_manager.cc:1347] Generated new cluster ID: 99445e6b1d8f4ea2a09072affc782a77
I20260927 03:17:59.218357 20100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:17:59.222453 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.210383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.17.52.190:45155)
0927 03:17:59.210652 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:59.210665 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:59.210739 (+ 74us) server_negotiation.cc:407] Connection header received
0927 03:17:59.211290 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:59.211297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:59.211348 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:59.211462 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:59.212268 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.213686 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.214842 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.215457 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.215561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.218580 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:59.218585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:59.218585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:59.218598 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:59.220676 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.221039 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.221042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.221043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.221151 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.221350 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.221351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.221351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.221655 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:59.221825 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:59.221923 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:17:59.222059 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260927 03:17:59.233088 20100 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:17:59.233666 20100 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:17:59.239430 20100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7988c62bd6ce4778bd01bce4f7411311: Generated new TSK 0
I20260927 03:17:59.239629 20100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:17:59.267081 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:59.349035 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:59.349255 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:59.349279 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:59.350503 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:59.350549 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:59.350576 20106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:17:59.352116 20106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:59.352384 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:59.352695 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:59.357605 20115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.357605 20113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.357676 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:59.357956 20106 server_base.cc:1034] running on GCE node
I20260927 03:17:59.358160 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:59.358418 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:59.359570 20106 hybrid_clock.cc:648] HybridClock initialized: now 1790479079359555 us; error 28 us; skew 500 ppm
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:59.362181 20106 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:17:59.362627 20106 webserver.cc:460] Webserver started at http://127.17.52.129:45087/ using document root <none> and password file <none>
I20260927 03:17:59.362875 20106 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:59.362947 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:59.363093 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:59.364137 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "c4520dc4834641f08ab6c0cd5db182fe"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "8c666f7d9836132bc2f6d69d81bfa326"
server_key_iv: "fbadc883ba047520d0d9709c480cb760"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.364502 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "c4520dc4834641f08ab6c0cd5db182fe"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "8c666f7d9836132bc2f6d69d81bfa326"
server_key_iv: "fbadc883ba047520d0d9709c480cb760"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.365823 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:17:59.366533 20122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.366703 20106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.366799 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "c4520dc4834641f08ab6c0cd5db182fe"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "8c666f7d9836132bc2f6d69d81bfa326"
server_key_iv: "fbadc883ba047520d0d9709c480cb760"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.366883 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:59.375154 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:59.377482 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:59.377657 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:59.377897 20106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:59.378213 20106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:59.378268 20106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.378304 20106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:59.378330 20106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.383982 20106 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:34499
I20260927 03:17:59.384022 20235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:34499 every 8 connection(s)
I20260927 03:17:59.384349 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:59.391757 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20106
I20260927 03:17:59.391860 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:17:59.392064 17618 external_mini_cluster.cc:1428] Setting key a64c4557b21c3901e8dcfcb7ab95890c
I20260927 03:17:59.393980 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.385083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:51867 (local address 127.17.52.190:45155)
0927 03:17:59.385128 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:59.385130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:59.385419 (+ 289us) server_negotiation.cc:407] Connection header received
0927 03:17:59.385929 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:59.385930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:59.385940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:59.385993 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:59.386934 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.387449 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.388457 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.388722 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.388787 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.390901 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:59.390903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:59.390904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:59.390912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:59.392529 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.392984 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.392986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.392986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.393045 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.393308 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.393309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.393310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.393431 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:59.393507 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:59.393801 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 03:17:59.393886 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:17:59.394232 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.385079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45155 (local address 127.17.52.129:51867)
0927 03:17:59.385252 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:59.385313 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:59.385829 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:59.386075 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:59.386078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:59.386650 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:59.386850 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.386855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.387538 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.387539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.388374 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.388376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.388820 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.388821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.388940 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.389398 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:59.389406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:59.390795 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:59.392660 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.392666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.392668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.392905 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.393131 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.393132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.393132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.393240 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.393498 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:59.393504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:59.393641 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:17:59.393780 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 03:17:59.393944 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260927 03:17:59.394724 20236 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45155
I20260927 03:17:59.394819 20236 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:59.395013 20236 heartbeater.cc:511] Master 127.17.52.190:45155 requested a full tablet report, sending...
I20260927 03:17:59.395552 20044 ts_manager.cc:194] Registered new tserver with Master: c4520dc4834641f08ab6c0cd5db182fe (127.17.52.129:34499)
I20260927 03:17:59.396157 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:51867
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:17:59.416910 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:59.499761 20241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:59.499969 20241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:59.499991 20241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:59.501099 20241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:59.501134 20241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:59.501159 20241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:17:59.502518 20241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:59.502708 20241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:59.502985 20241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:59.507751 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:59.507828 20241 server_base.cc:1034] running on GCE node
W20260927 03:17:59.507709 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.507714 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:59.508147 20241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:59.508399 20241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:59.509572 20241 hybrid_clock.cc:648] HybridClock initialized: now 1790479079509547 us; error 51 us; skew 500 ppm
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:59.511967 20241 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:17:59.512341 20241 webserver.cc:460] Webserver started at http://127.17.52.130:35821/ using document root <none> and password file <none>
I20260927 03:17:59.512529 20241 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:59.512619 20241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:59.512759 20241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:59.513875 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "b34f199b8d2445a6ab3c9b143647437b"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "c751154a02bf67a7c364f5e52eb3c481"
server_key_iv: "3a9e8381f67ba7c33a789392f1dd62d1"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.514286 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "b34f199b8d2445a6ab3c9b143647437b"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "c751154a02bf67a7c364f5e52eb3c481"
server_key_iv: "3a9e8381f67ba7c33a789392f1dd62d1"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.515698 20241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:17:59.516405 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.516600 20241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:17:59.516712 20241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "b34f199b8d2445a6ab3c9b143647437b"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "c751154a02bf67a7c364f5e52eb3c481"
server_key_iv: "3a9e8381f67ba7c33a789392f1dd62d1"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.516767 20241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:59.527706 20241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:59.529598 20241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:59.529732 20241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:59.529906 20241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:59.530203 20241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:59.530236 20241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.530257 20241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:59.530304 20241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.535827 20241 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:35471
I20260927 03:17:59.535907 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:35471 every 8 connection(s)
I20260927 03:17:59.536193 20241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:59.543442 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20241
I20260927 03:17:59.543520 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:17:59.543694 17618 external_mini_cluster.cc:1428] Setting key ed7b3f6028954d8de94edfcf0499eeab
I20260927 03:17:59.545339 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.536908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45155 (local address 127.17.52.130:55587)
0927 03:17:59.537067 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:59.537108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:59.537564 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:59.537811 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:59.537814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:59.538361 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:59.538543 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.538547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.539222 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.539223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.540029 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.540031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.540347 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.540348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.540448 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.540952 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:59.540960 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:59.542347 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:59.543825 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.543831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.543833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.544121 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.544389 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.544390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.544390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.544528 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.544874 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:59.544879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:59.544988 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 03:17:59.545069 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:17:59.545182 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 03:17:59.545339 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.536949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:55587 (local address 127.17.52.190:45155)
0927 03:17:59.536991 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:59.536993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:59.537177 (+ 184us) server_negotiation.cc:407] Connection header received
0927 03:17:59.537670 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:59.537671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:59.537679 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:59.537737 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:59.538639 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.539142 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.540103 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.540280 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.540329 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.542441 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:59.542443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:59.542443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:59.542449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:59.543716 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.544213 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.544214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.544214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.544310 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.544623 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.544624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.544625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.544806 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:59.544881 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:59.545112 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:17:59.545203 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:17:59.545801 20371 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45155
I20260927 03:17:59.545918 20371 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:59.546082 20371 heartbeater.cc:511] Master 127.17.52.190:45155 requested a full tablet report, sending...
I20260927 03:17:59.546469 20044 ts_manager.cc:194] Registered new tserver with Master: b34f199b8d2445a6ab3c9b143647437b (127.17.52.130:35471)
I20260927 03:17:59.546980 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:55587
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:17:59.567639 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:17:59.651980 20376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:17:59.652161 20376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:17:59.652184 20376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:17:59.653380 20376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:17:59.653422 20376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:17:59.653448 20376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:17:59.654850 20376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:17:59.655069 20376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:17:59.655287 20376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:17:59.660048 20382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.660048 20381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:17:59.660076 20384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:17:59.660295 20376 server_base.cc:1034] running on GCE node
I20260927 03:17:59.660506 20376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:17:59.660799 20376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:17:59.661949 20376 hybrid_clock.cc:648] HybridClock initialized: now 1790479079661941 us; error 32 us; skew 500 ppm
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:17:59.664336 20376 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:17:59.664806 20376 webserver.cc:460] Webserver started at http://127.17.52.131:40427/ using document root <none> and password file <none>
I20260927 03:17:59.665035 20376 fs_manager.cc:359] Metadata directory not provided
I20260927 03:17:59.665078 20376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:17:59.665172 20376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:17:59.666091 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "d096d904154f4492ac2a8f79563326b2"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "cfda0f4ac101fa919c2b4198b4d6fac3"
server_key_iv: "6a91df36aa176ecf7873374f4afa1ca8"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.666388 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "d096d904154f4492ac2a8f79563326b2"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "cfda0f4ac101fa919c2b4198b4d6fac3"
server_key_iv: "6a91df36aa176ecf7873374f4afa1ca8"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.667586 20376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:17:59.668241 20391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.668407 20376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:17:59.668489 20376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "d096d904154f4492ac2a8f79563326b2"
format_stamp: "Formatted at 2026-09-27 03:17:59 on dist-test-slave-7j43"
server_key: "cfda0f4ac101fa919c2b4198b4d6fac3"
server_key_iv: "6a91df36aa176ecf7873374f4afa1ca8"
server_key_version: "encryptionkey@0"
I20260927 03:17:59.668540 20376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:17:59.682233 20376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:17:59.683898 20376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:17:59.684001 20376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:17:59.684165 20376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:17:59.684463 20376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:17:59.684494 20376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.684514 20376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:17:59.684561 20376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:17:59.690713 20376 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42969
I20260927 03:17:59.690774 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42969 every 8 connection(s)
I20260927 03:17:59.691198 20376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:17:59.692523 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20376
I20260927 03:17:59.692736 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:17:59.692989 17618 external_mini_cluster.cc:1428] Setting key e5f02560eb2bd0bbb6016bb29efcd0e9
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479079, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:17:59.701483 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.692114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:57785 (local address 127.17.52.190:45155)
0927 03:17:59.692161 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:59.692164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:59.692423 (+ 259us) server_negotiation.cc:407] Connection header received
0927 03:17:59.693200 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:59.693202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:59.693217 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:59.693250 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:59.694593 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.695159 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.696479 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.696815 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.696885 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.698985 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:17:59.698986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:17:59.698987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:17:59.698996 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:17:59.700146 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.700537 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.700538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.700538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.700614 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:17:59.700842 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:17:59.700843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:17:59.700844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:17:59.700962 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:17:59.701041 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:17:59.701314 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 03:17:59.701392 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:17:59.701673 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:17:59.691967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45155 (local address 127.17.52.131:57785)
0927 03:17:59.692300 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 03:17:59.692342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:17:59.693068 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:17:59.693454 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:17:59.693459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:17:59.694209 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:17:59.694505 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.694512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.695250 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.695251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.696396 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:17:59.696399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:17:59.696904 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:17:59.696906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:17:59.697102 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:17:59.697601 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:17:59.697609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:17:59.698892 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:17:59.700229 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.700235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.700237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.700461 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.700664 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:17:59.700665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:17:59.700665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:17:59.700774 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:17:59.701029 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:17:59.701034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:17:59.701170 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:17:59.701289 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:17:59.701435 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":79,"threads_started":1}
I20260927 03:17:59.702071 20506 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45155
I20260927 03:17:59.702153 20506 heartbeater.cc:464] Registering TS with master...
I20260927 03:17:59.702363 20506 heartbeater.cc:511] Master 127.17.52.190:45155 requested a full tablet report, sending...
I20260927 03:17:59.702690 20044 ts_manager.cc:194] Registered new tserver with Master: d096d904154f4492ac2a8f79563326b2 (127.17.52.131:42969)
I20260927 03:17:59.703017 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:57785
I20260927 03:17:59.704782 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:17:59.715394 20084 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:17:59.714504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.17.52.190:45155)
0927 03:17:59.714563 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:17:59.714565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:17:59.714685 (+ 120us) server_negotiation.cc:407] Connection header received
0927 03:17:59.714727 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:17:59.714728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:17:59.714737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:17:59.714792 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:17:59.715315 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47454: BlockingRecv error: recv got EOF from 127.0.0.1:47454 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:17:59.716922 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20106
I20260927 03:17:59.721820 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20241
I20260927 03:17:59.725971 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20376
I20260927 03:17:59.729938 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20012
2026-09-27T03:17:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1816 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[20518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[20518](info): set up 2 sockets
Sep 27 03:17:59 dist-test-slave-7j43 krb5kdc[20518](info): commencing operation
krb5kdc: starting...
W20260927 03:18:00.764937 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:00 dist-test-slave-7j43 krb5kdc[20518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:00.879909 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:36479
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:00.960913 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:00.961109 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:00.961133 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:00.962296 20531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:00.962338 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:00.962353 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:00.962366 20531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:00.962378 20531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:00.963850 20531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:00.964100 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:00.964327 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:18:00.969465 20531 server_base.cc:1034] running on GCE node
W20260927 03:18:00.969475 20537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:00.969550 20540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:00.969482 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:00.969938 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:00.970227 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:00.971374 20531 hybrid_clock.cc:648] HybridClock initialized: now 1790479080971364 us; error 31 us; skew 500 ppm
Sep 27 03:18:00 dist-test-slave-7j43 krb5kdc[20518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479080, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:00.973810 20531 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:00.974272 20531 webserver.cc:460] Webserver started at http://127.17.52.190:46507/ using document root <none> and password file <none>
I20260927 03:18:00.974500 20531 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:00.974581 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:00.974731 20531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:00.975816 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "bb71d8585c5c48638de9814a61ad3e72"
format_stamp: "Formatted at 2026-09-27 03:18:00 on dist-test-slave-7j43"
server_key: "1a8d8e593781624e3b334f951b3dffc7"
server_key_iv: "3287733a1d7bd386c498156901397eb9"
server_key_version: "encryptionkey@0"
I20260927 03:18:00.976234 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "bb71d8585c5c48638de9814a61ad3e72"
format_stamp: "Formatted at 2026-09-27 03:18:00 on dist-test-slave-7j43"
server_key: "1a8d8e593781624e3b334f951b3dffc7"
server_key_iv: "3287733a1d7bd386c498156901397eb9"
server_key_version: "encryptionkey@0"
I20260927 03:18:00.977618 20531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:00.978338 20547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:00.978516 20531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:00.978621 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "bb71d8585c5c48638de9814a61ad3e72"
format_stamp: "Formatted at 2026-09-27 03:18:00 on dist-test-slave-7j43"
server_key: "1a8d8e593781624e3b334f951b3dffc7"
server_key_iv: "3287733a1d7bd386c498156901397eb9"
server_key_version: "encryptionkey@0"
I20260927 03:18:00.978693 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:00.987322 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:00.989244 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:00.989436 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:00.993609 20531 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:36479
I20260927 03:18:00.993680 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:36479 every 8 connection(s)
I20260927 03:18:00.994164 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:00.994743 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20531
I20260927 03:18:00.994827 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:00.994866 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:00.995036 17618 external_mini_cluster.cc:1428] Setting key 30a7a4731dab4864111965bf3117d5ed
I20260927 03:18:00.998296 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72: Bootstrap starting.
I20260927 03:18:00.998982 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:00.999271 20600 log.cc:824] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:00.999925 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72: No bootstrap required, opened a new log
Sep 27 03:18:00 dist-test-slave-7j43 krb5kdc[20518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:01.002552 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } }
I20260927 03:18:01.002713 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:01.002787 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb71d8585c5c48638de9814a61ad3e72, State: Initialized, Role: FOLLOWER
I20260927 03:18:01.002935 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } }
I20260927 03:18:01.003026 20600 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:01.003060 20600 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:01.003100 20600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:01.003751 20600 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } }
I20260927 03:18:01.003891 20600 leader_election.cc:302] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb71d8585c5c48638de9814a61ad3e72; no voters:
I20260927 03:18:01.004063 20600 leader_election.cc:288] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:01.004094 20605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:01.004206 20605 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [term 1 LEADER]: Becoming Leader. State: Replica: bb71d8585c5c48638de9814a61ad3e72, State: Running, Role: LEADER
I20260927 03:18:01.004289 20605 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } }
I20260927 03:18:01.004355 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:01.004653 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb71d8585c5c48638de9814a61ad3e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } } }
I20260927 03:18:01.004726 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:01.004858 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb71d8585c5c48638de9814a61ad3e72. Latest consensus state: current_term: 1 leader_uuid: "bb71d8585c5c48638de9814a61ad3e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d8585c5c48638de9814a61ad3e72" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36479 } } }
I20260927 03:18:01.004927 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:01.005137 20610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:01.005641 20610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:01.006671 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:00.995507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.17.52.190:36479)
0927 03:18:00.995803 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:00.995813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:00.995846 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:18:00.996495 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:00.996505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:00.996571 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:00.996749 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:00.997314 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:00.998708 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:00.999519 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:00.999948 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:01.000073 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:01.002696 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:01.002700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:01.002701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:01.002711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:01.004825 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:01.005240 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:01.005243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:01.005243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:01.005345 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:01.005554 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:01.005555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:01.005556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:01.005897 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:01.006044 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:01.006305 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:18:01.006464 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":155,"threads_started":1}
I20260927 03:18:01.007217 20610 catalog_manager.cc:1347] Generated new cluster ID: cfa3535894ea429e9efae9a955e522cf
I20260927 03:18:01.007268 20610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:01.007560 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:18:01.015993 20610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:01.016804 20610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:01.023528 20610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb71d8585c5c48638de9814a61ad3e72: Generated new TSK 0
I20260927 03:18:01.023727 20610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:01.029695 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:01.025328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.17.52.190:36479)
0927 03:18:01.025391 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:01.025393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:01.025538 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:18:01.025607 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:01.025608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:01.025617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:01.025663 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:01.026122 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:01.026986 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:01.028115 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:01.028333 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:01.028389 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:01.029201 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:01.029202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:01.029203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:01.029417 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:01.029434 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:01.029441 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:01.029489 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:01.029575 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:18:01.029600 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:18:01.031880 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20531
2026-09-27T03:18:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1300 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:01 dist-test-slave-7j43 krb5kdc[20633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:01 dist-test-slave-7j43 krb5kdc[20633](info): set up 2 sockets
Sep 27 03:18:01 dist-test-slave-7j43 krb5kdc[20633](info): commencing operation
krb5kdc: starting...
W20260927 03:18:02.061010 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:02.177734 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:32971
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:02.259173 20646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:02.259430 20646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:02.259469 20646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:02.260728 20646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:02.260804 20646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:02.260829 20646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:02.260851 20646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:02.260878 20646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:02.262382 20646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:02.262668 20646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:02.262898 20646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:02.267851 20654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.267853 20652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.267907 20651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:02.268136 20646 server_base.cc:1034] running on GCE node
I20260927 03:18:02.268494 20646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:02.268855 20646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:02.270027 20646 hybrid_clock.cc:648] HybridClock initialized: now 1790479082270005 us; error 36 us; skew 500 ppm
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:02.272750 20646 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:02.273231 20646 webserver.cc:460] Webserver started at http://127.17.52.190:38161/ using document root <none> and password file <none>
I20260927 03:18:02.273492 20646 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:02.273540 20646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:02.273685 20646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:02.274803 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "b6ef46d4437d421e8462dead7ba963b4"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "dfc869c52b5103e7a67215477989d349"
server_key_iv: "65def1c080fa654a3b85863f422b75f3"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.275166 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "b6ef46d4437d421e8462dead7ba963b4"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "dfc869c52b5103e7a67215477989d349"
server_key_iv: "65def1c080fa654a3b85863f422b75f3"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.276554 20646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:02.277422 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.277639 20646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:02.277753 20646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "b6ef46d4437d421e8462dead7ba963b4"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "dfc869c52b5103e7a67215477989d349"
server_key_iv: "65def1c080fa654a3b85863f422b75f3"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.277833 20646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:02.297936 20646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:02.300048 20646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:02.300218 20646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:02.304448 20646 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:32971
I20260927 03:18:02.304484 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:32971 every 8 connection(s)
I20260927 03:18:02.304868 20646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:02.305877 20714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:02.308166 20714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4: Bootstrap starting.
I20260927 03:18:02.308741 20714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:02.308977 20714 log.cc:824] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:02.309652 20714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4: No bootstrap required, opened a new log
I20260927 03:18:02.311471 20714 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } }
I20260927 03:18:02.311627 20714 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:02.311654 20714 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ef46d4437d421e8462dead7ba963b4, State: Initialized, Role: FOLLOWER
I20260927 03:18:02.311812 20714 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } }
I20260927 03:18:02.311908 20714 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:02.311935 20714 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:02.311973 20714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:02.312852 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20646
I20260927 03:18:02.312968 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:02.312953 20714 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } }
I20260927 03:18:02.313059 20714 leader_election.cc:302] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ef46d4437d421e8462dead7ba963b4; no voters:
I20260927 03:18:02.313182 17618 external_mini_cluster.cc:1428] Setting key f5e243ef017b29cd8c583f6d53a3f963
I20260927 03:18:02.313239 20714 leader_election.cc:288] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:02.313315 20717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:02.313488 20717 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [term 1 LEADER]: Becoming Leader. State: Replica: b6ef46d4437d421e8462dead7ba963b4, State: Running, Role: LEADER
I20260927 03:18:02.313570 20714 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:02.313606 20717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } }
I20260927 03:18:02.314291 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ef46d4437d421e8462dead7ba963b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } } }
I20260927 03:18:02.314385 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:02.314548 20719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ef46d4437d421e8462dead7ba963b4. Latest consensus state: current_term: 1 leader_uuid: "b6ef46d4437d421e8462dead7ba963b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ef46d4437d421e8462dead7ba963b4" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 32971 } } }
I20260927 03:18:02.314631 20719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:02.314983 20728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:02.315492 20728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:02.317369 20728 catalog_manager.cc:1347] Generated new cluster ID: f0838561d1d64ccbb89fe698beeb991d
I20260927 03:18:02.317430 20728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:02.325253 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.313728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.17.52.190:32971)
0927 03:18:02.314014 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:02.314025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:02.314080 (+ 55us) server_negotiation.cc:407] Connection header received
0927 03:18:02.314221 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:02.314230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:02.314298 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:02.315159 (+ 861us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:02.315432 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.316827 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.317914 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.318529 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.318646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.321506 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:02.321511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:02.321511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:02.321523 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:02.323529 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.323946 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.323949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.323949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.324088 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.324307 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.324307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.324308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.324624 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:02.324782 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:02.324893 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:18:02.325028 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
I20260927 03:18:02.342576 20728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:02.343370 20728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:02.347649 20728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ef46d4437d421e8462dead7ba963b4: Generated new TSK 0
I20260927 03:18:02.347851 20728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:02.373250 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:02.455259 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:02.455506 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:02.455547 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:02.456772 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:02.456837 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:02.456882 20740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:02.458372 20740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:02.458640 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:02.458879 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:02.464449 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.464445 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.464445 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:02.464608 20740 server_base.cc:1034] running on GCE node
I20260927 03:18:02.464956 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:02.465212 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:02.466373 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790479082466359 us; error 29 us; skew 500 ppm
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:02.469055 20740 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:02.469542 20740 webserver.cc:460] Webserver started at http://127.17.52.129:36277/ using document root <none> and password file <none>
I20260927 03:18:02.469779 20740 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:02.469847 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:02.469982 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:02.471071 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "8381d6821f464f1db1a5af32368a2412"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "4f0f5064adf94a53f84d27e843b07e56"
server_key_iv: "f0405d25ea12c26180a88a5b77adc5df"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.471450 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "8381d6821f464f1db1a5af32368a2412"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "4f0f5064adf94a53f84d27e843b07e56"
server_key_iv: "f0405d25ea12c26180a88a5b77adc5df"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.472968 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:02.473896 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.474138 20740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:02.474252 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "8381d6821f464f1db1a5af32368a2412"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "4f0f5064adf94a53f84d27e843b07e56"
server_key_iv: "f0405d25ea12c26180a88a5b77adc5df"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.474355 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:02.492268 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:02.494454 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:02.494632 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:02.494885 20740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:02.495232 20740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:02.495296 20740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.495342 20740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:02.495370 20740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.501161 20740 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:44051
I20260927 03:18:02.501217 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:44051 every 8 connection(s)
I20260927 03:18:02.501546 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:02.508849 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20740
I20260927 03:18:02.508963 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:02.509178 17618 external_mini_cluster.cc:1428] Setting key 65257a4e87d36079d2670dc2699a547c
I20260927 03:18:02.511739 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.502309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50527 (local address 127.17.52.190:32971)
0927 03:18:02.502384 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:02.502387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:02.502653 (+ 266us) server_negotiation.cc:407] Connection header received
0927 03:18:02.503157 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:02.503158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:02.503171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:02.503227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:02.504169 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.504763 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.505758 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.505961 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.506013 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.508113 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:02.508115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:02.508116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:02.508124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:02.509976 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.510530 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.510531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.510532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.510588 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.510921 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.510921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.510922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.511034 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:02.511089 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:02.511461 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 03:18:02.511574 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:18:02.511833 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.502289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:32971 (local address 127.17.52.129:50527)
0927 03:18:02.502480 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:02.502546 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:02.503062 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:02.503306 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:02.503310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:02.503893 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:02.504086 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.504090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.504847 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.504848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.505672 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.505674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.506034 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.506035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.506137 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.506617 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:02.506625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:02.508019 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:02.510081 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.510089 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.510091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.510446 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.510690 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.510691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.510691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.510849 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.511118 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:02.511124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:02.511305 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 03:18:02.511424 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:18:02.511628 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260927 03:18:02.512354 20869 heartbeater.cc:347] Connected to a master server at 127.17.52.190:32971
I20260927 03:18:02.512468 20869 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:02.512687 20869 heartbeater.cc:511] Master 127.17.52.190:32971 requested a full tablet report, sending...
I20260927 03:18:02.513219 20678 ts_manager.cc:194] Registered new tserver with Master: 8381d6821f464f1db1a5af32368a2412 (127.17.52.129:44051)
I20260927 03:18:02.513934 20678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:50527
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:02.533391 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:02.617091 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:02.617326 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:02.617367 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:02.618562 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:02.618626 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:02.618672 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:02.620146 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:02.620411 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:02.620690 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:02.625880 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.625890 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.625888 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:02.626010 20874 server_base.cc:1034] running on GCE node
I20260927 03:18:02.626271 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:02.626542 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:02.627715 20874 hybrid_clock.cc:648] HybridClock initialized: now 1790479082627709 us; error 35 us; skew 500 ppm
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:02.630100 20874 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:02.630553 20874 webserver.cc:460] Webserver started at http://127.17.52.130:38435/ using document root <none> and password file <none>
I20260927 03:18:02.630784 20874 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:02.630852 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:02.630988 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:02.632036 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "cf7b4b7515124c54b17e11569839a554"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8a6aeb5e76916d1d1ec8df6355d42902"
server_key_iv: "73652fe0f42a6a97218693f009d76242"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.632398 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "cf7b4b7515124c54b17e11569839a554"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8a6aeb5e76916d1d1ec8df6355d42902"
server_key_iv: "73652fe0f42a6a97218693f009d76242"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.633733 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:02.634452 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.634625 20874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:02.634718 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "cf7b4b7515124c54b17e11569839a554"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8a6aeb5e76916d1d1ec8df6355d42902"
server_key_iv: "73652fe0f42a6a97218693f009d76242"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.634817 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:02.641126 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:02.642916 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:02.643070 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:02.643282 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:02.643579 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:02.643632 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.643672 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:02.643702 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.649955 20874 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:40567
I20260927 03:18:02.650002 21003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:40567 every 8 connection(s)
I20260927 03:18:02.650328 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:02.658285 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 20874
I20260927 03:18:02.658396 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:02.658638 17618 external_mini_cluster.cc:1428] Setting key a040c1745cbb473734e2f5497ffe0328
I20260927 03:18:02.659621 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.651087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:53407 (local address 127.17.52.190:32971)
0927 03:18:02.651155 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:02.651157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:02.651377 (+ 220us) server_negotiation.cc:407] Connection header received
0927 03:18:02.651839 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:02.651840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:02.651851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:02.651905 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:02.652878 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.653384 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.654321 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.654508 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.654559 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.656684 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:02.656685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:02.656686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:02.656692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:02.658086 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.658584 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.658585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.658585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.658638 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.658995 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.658996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.658996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.659103 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:02.659158 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:02.659488 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 03:18:02.659555 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:18:02.659941 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.651072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:32971 (local address 127.17.52.130:53407)
0927 03:18:02.651249 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:02.651289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:02.651748 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:02.651980 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:02.651983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:02.652532 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:02.652795 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.652800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.653463 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.653464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.654239 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.654241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.654576 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.654577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.654672 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.655139 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:02.655147 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:02.656528 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:02.658159 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.658165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.658167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.658508 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.658707 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.658708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.658708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.658849 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.659170 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:02.659176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:02.659347 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:18:02.659468 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:18:02.659661 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260927 03:18:02.660526 21004 heartbeater.cc:347] Connected to a master server at 127.17.52.190:32971
I20260927 03:18:02.660651 21004 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:02.660859 21004 heartbeater.cc:511] Master 127.17.52.190:32971 requested a full tablet report, sending...
I20260927 03:18:02.661286 20678 ts_manager.cc:194] Registered new tserver with Master: cf7b4b7515124c54b17e11569839a554 (127.17.52.130:40567)
I20260927 03:18:02.661814 20678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:53407
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:02.681324 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:02.761598 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:02.761802 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:02.761823 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:02.762950 21009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:02.762991 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:02.763017 21009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:02.764443 21009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:02.764724 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:02.764945 21009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:02.769749 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:02.769758 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:02.769855 21009 server_base.cc:1034] running on GCE node
W20260927 03:18:02.769761 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:02.770190 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:02.770442 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:02.771624 21009 hybrid_clock.cc:648] HybridClock initialized: now 1790479082771612 us; error 28 us; skew 500 ppm
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:02.774209 21009 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:02.774665 21009 webserver.cc:460] Webserver started at http://127.17.52.131:40351/ using document root <none> and password file <none>
I20260927 03:18:02.774866 21009 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:02.774909 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:02.774995 21009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:02.775965 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "c5f89694e8a0496582991a8052011798"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8b89d1d612afa5c35a7c96921ff4da27"
server_key_iv: "37bc3bfa5944e43c0c1bb585b9752f1e"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.776265 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "c5f89694e8a0496582991a8052011798"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8b89d1d612afa5c35a7c96921ff4da27"
server_key_iv: "37bc3bfa5944e43c0c1bb585b9752f1e"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.777688 21009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:02.778395 21024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.778578 21009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:02.778650 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "c5f89694e8a0496582991a8052011798"
format_stamp: "Formatted at 2026-09-27 03:18:02 on dist-test-slave-7j43"
server_key: "8b89d1d612afa5c35a7c96921ff4da27"
server_key_iv: "37bc3bfa5944e43c0c1bb585b9752f1e"
server_key_version: "encryptionkey@0"
I20260927 03:18:02.778708 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:02.787415 21009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:02.789613 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:02.789721 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:02.789929 21009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:02.790203 21009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:02.790235 21009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.790258 21009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:02.790272 21009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:02.796243 21009 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:37817
I20260927 03:18:02.796298 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:37817 every 8 connection(s)
I20260927 03:18:02.796715 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:02.806137 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21009
I20260927 03:18:02.806152 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.797462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:45637 (local address 127.17.52.190:32971)
0927 03:18:02.797528 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:02.797530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:02.797744 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:18:02.798266 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:02.798267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:02.798280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:02.798336 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:02.799274 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.799878 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.800949 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.801154 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.801239 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.803330 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:02.803332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:02.803332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:02.803340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:02.804801 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.805212 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.805213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.805214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.805269 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.805523 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.805524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.805524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.805636 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:02.805707 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:02.805985 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:18:02.806061 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:18:02.806274 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:02.806334 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.797459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:32971 (local address 127.17.52.131:45637)
0927 03:18:02.797620 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:02.797656 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:02.798158 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:02.798412 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:02.798416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:02.798986 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:02.799173 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.799178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.799974 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.799975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.800850 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:02.800853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.801248 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.801249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:02.801374 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.801908 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:02.801916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:02.803234 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:02.804896 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.804902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.804905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.805123 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.805346 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:02.805346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:02.805347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:02.805450 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:02.805717 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:02.805723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:02.805855 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 03:18:02.805962 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:18:02.806114 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260927 03:18:02.806469 17618 external_mini_cluster.cc:1428] Setting key a1a3fbfc38858fe97056bcb835def00d
I20260927 03:18:02.806962 21139 heartbeater.cc:347] Connected to a master server at 127.17.52.190:32971
I20260927 03:18:02.807080 21139 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:02.807294 21139 heartbeater.cc:511] Master 127.17.52.190:32971 requested a full tablet report, sending...
I20260927 03:18:02.807687 20678 ts_manager.cc:194] Registered new tserver with Master: c5f89694e8a0496582991a8052011798 (127.17.52.131:37817)
I20260927 03:18:02.808007 20678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:45637
I20260927 03:18:02.818256 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479082, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:02.840416 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:02.833661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.17.52.190:32971)
0927 03:18:02.833741 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:02.833743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:02.833797 (+ 54us) server_negotiation.cc:407] Connection header received
0927 03:18:02.833857 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:02.833858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:02.833867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:02.833917 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:02.834408 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.834895 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.835568 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:02.835747 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:02.835795 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:02.838050 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:02.838052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:02.838052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:02.838059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:02.839478 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.839759 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.839760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.839760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.839808 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:02.839948 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:02.839948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:02.839948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:02.840080 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:02.840149 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:02.840228 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:18:02.840300 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 03:18:02.840687 20678 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57406
I20260927 03:18:02.841188 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20740
I20260927 03:18:02.845789 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20874
I20260927 03:18:02.850145 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21009
I20260927 03:18:02.854323 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 20646
2026-09-27T03:18:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1824 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[21151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[21151](info): set up 2 sockets
Sep 27 03:18:02 dist-test-slave-7j43 krb5kdc[21151](info): commencing operation
krb5kdc: starting...
W20260927 03:18:03.892974 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:03 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:04.007176 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42897
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:42897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:04.086545 21164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:04.086786 21164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:04.086825 21164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:04.087957 21164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:04.088022 21164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:04.088047 21164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:04.088073 21164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:04.088104 21164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:04.089608 21164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:42897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:04.089877 21164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:04.090101 21164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:18:04.094997 21164 server_base.cc:1034] running on GCE node
W20260927 03:18:04.094981 21170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.095019 21172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.094960 21169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:04.095384 21164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:04.095700 21164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:04.096838 21164 hybrid_clock.cc:648] HybridClock initialized: now 1790479084096829 us; error 30 us; skew 500 ppm
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:04.099152 21164 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:04.099512 21164 webserver.cc:460] Webserver started at http://127.17.52.190:35075/ using document root <none> and password file <none>
I20260927 03:18:04.099684 21164 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:04.099722 21164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:04.099814 21164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:04.100867 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "a7f7ef954db3490190e1dae22788d5e2"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "35492f486529a23bfb30c605acfe5e21"
server_key_iv: "e2c029d66fa7f496d0da4ab40b47be08"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.101214 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "a7f7ef954db3490190e1dae22788d5e2"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "35492f486529a23bfb30c605acfe5e21"
server_key_iv: "e2c029d66fa7f496d0da4ab40b47be08"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.102448 21164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:04.103240 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.103417 21164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:04.103524 21164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "a7f7ef954db3490190e1dae22788d5e2"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "35492f486529a23bfb30c605acfe5e21"
server_key_iv: "e2c029d66fa7f496d0da4ab40b47be08"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.103596 21164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:04.119143 21164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:04.120877 21164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:04.121033 21164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:04.124833 21164 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:42897
I20260927 03:18:04.124886 21232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:42897 every 8 connection(s)
I20260927 03:18:04.125391 21164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:04.126035 21233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:04.128218 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2: Bootstrap starting.
I20260927 03:18:04.128847 21233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:04.129148 21233 log.cc:824] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:04.129825 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2: No bootstrap required, opened a new log
I20260927 03:18:04.131415 21233 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } }
I20260927 03:18:04.131554 21233 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:04.131577 21233 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f7ef954db3490190e1dae22788d5e2, State: Initialized, Role: FOLLOWER
I20260927 03:18:04.131676 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21164
I20260927 03:18:04.131740 21233 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } }
I20260927 03:18:04.131790 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:04.131829 21233 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:04.131877 21233 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:04.131935 21233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:04.131974 17618 external_mini_cluster.cc:1428] Setting key 1f6305624f038811d11aec2f86d4740b
I20260927 03:18:04.133168 21233 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } }
I20260927 03:18:04.133301 21233 leader_election.cc:302] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f7ef954db3490190e1dae22788d5e2; no voters:
I20260927 03:18:04.133477 21233 leader_election.cc:288] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:04.133535 21238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:04.133724 21238 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [term 1 LEADER]: Becoming Leader. State: Replica: a7f7ef954db3490190e1dae22788d5e2, State: Running, Role: LEADER
I20260927 03:18:04.133747 21233 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:04.133827 21238 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } }
I20260927 03:18:04.134150 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f7ef954db3490190e1dae22788d5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } } }
I20260927 03:18:04.134298 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:04.134464 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f7ef954db3490190e1dae22788d5e2. Latest consensus state: current_term: 1 leader_uuid: "a7f7ef954db3490190e1dae22788d5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f7ef954db3490190e1dae22788d5e2" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42897 } } }
I20260927 03:18:04.134570 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:04.134927 21246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:04.135407 21246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:04.136905 21246 catalog_manager.cc:1347] Generated new cluster ID: a17959010c7743d5aef75a6e97eb5e57
I20260927 03:18:04.136956 21246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:04.143461 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.132456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.17.52.190:42897)
0927 03:18:04.132660 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.132668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.132693 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:18:04.132791 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.132797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.132838 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.132946 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.133785 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.135196 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.136360 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.136987 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.137095 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.139766 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.139770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.139771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.139784 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.141794 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.142181 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.142185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.142185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.142336 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.142545 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.142546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.142546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.142881 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.143056 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.143123 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.143262 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260927 03:18:04.153465 21246 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:04.154219 21246 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:04.162954 21246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f7ef954db3490190e1dae22788d5e2: Generated new TSK 0
I20260927 03:18:04.163175 21246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:04.195050 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:04.274698 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:04.274904 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:04.274927 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:04.276067 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:04.276105 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:04.276132 21259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:04.277645 21259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:04.277899 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:04.278129 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:04.283043 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:04.283116 21259 server_base.cc:1034] running on GCE node
W20260927 03:18:04.283074 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.283083 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:04.283442 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:04.283644 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:04.284869 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790479084284862 us; error 36 us; skew 500 ppm
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:04.290990 21259 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:04.291484 21259 webserver.cc:460] Webserver started at http://127.17.52.129:34251/ using document root <none> and password file <none>
I20260927 03:18:04.291688 21259 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:04.291728 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:04.291824 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:04.292881 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "edade4491f6a4810a7a4a4be725b9dda"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "75d87e07e5d8a3a89cae865f4062ec0b"
server_key_iv: "cfe01529a59395ee96ad2df937d2d7e2"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.293171 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "edade4491f6a4810a7a4a4be725b9dda"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "75d87e07e5d8a3a89cae865f4062ec0b"
server_key_iv: "cfe01529a59395ee96ad2df937d2d7e2"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.294411 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:04.295096 21274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.295281 21259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:04.295368 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "edade4491f6a4810a7a4a4be725b9dda"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "75d87e07e5d8a3a89cae865f4062ec0b"
server_key_iv: "cfe01529a59395ee96ad2df937d2d7e2"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.295420 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:04.317188 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:04.319545 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:04.319664 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:04.319866 21259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:04.320187 21259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:04.320223 21259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.320246 21259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:04.320297 21259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.325793 21259 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:38281
I20260927 03:18:04.325856 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:38281 every 8 connection(s)
I20260927 03:18:04.326184 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:04.329645 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21259
I20260927 03:18:04.329746 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:04.329998 17618 external_mini_cluster.cc:1428] Setting key 5ff2542dcff28982b684ac756a48c621
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:04.337039 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.326992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:39847 (local address 127.17.52.190:42897)
0927 03:18:04.327156 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.327158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.327349 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:18:04.328334 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.328335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.328358 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.328412 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.329129 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.329667 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.331201 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.331406 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.331463 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.334055 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.334056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.334057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.334065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.335623 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.336049 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.336050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.336051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.336110 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.336359 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.336360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.336360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.336477 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.336549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.336857 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.336945 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 03:18:04.337282 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.326957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42897 (local address 127.17.52.129:39847)
0927 03:18:04.327194 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:04.327248 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:04.328212 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:04.328484 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:04.328488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:04.328837 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:04.329030 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.329034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.329822 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.329823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.331117 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.331120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.331489 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.331490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.331643 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.332251 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:04.332259 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:04.333931 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:04.335704 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.335711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.335714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.335940 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.336175 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.336176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.336177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.336289 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.336554 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:04.336560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:04.336711 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:18:04.336824 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:18:04.336985 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260927 03:18:04.337781 21389 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42897
I20260927 03:18:04.337899 21389 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:04.338112 21389 heartbeater.cc:511] Master 127.17.52.190:42897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
I20260927 03:18:04.338667 21196 ts_manager.cc:194] Registered new tserver with Master: edade4491f6a4810a7a4a4be725b9dda (127.17.52.129:38281)
I20260927 03:18:04.339268 21196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:39847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:04.355600 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:04.438308 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:04.438558 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:04.438598 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:04.439832 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:04.439908 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:04.439957 21394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:04.441637 21394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:04.441936 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:04.442179 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:04.447883 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.447860 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.447939 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:04.448230 21394 server_base.cc:1034] running on GCE node
I20260927 03:18:04.448414 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:04.448689 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:04.449852 21394 hybrid_clock.cc:648] HybridClock initialized: now 1790479084449842 us; error 27 us; skew 500 ppm
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:04.452221 21394 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:04.452732 21394 webserver.cc:460] Webserver started at http://127.17.52.130:35771/ using document root <none> and password file <none>
I20260927 03:18:04.452976 21394 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:04.453044 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:04.453190 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:04.454224 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "6a806483ba354a66b9ac6799659b5cc4"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "1ad5382887f6b230a0d99dab83d77653"
server_key_iv: "e8952ccc6269d40282a393b38be8a73d"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.454598 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "6a806483ba354a66b9ac6799659b5cc4"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "1ad5382887f6b230a0d99dab83d77653"
server_key_iv: "e8952ccc6269d40282a393b38be8a73d"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.455928 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:04.456696 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.456876 21394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:18:04.456975 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "6a806483ba354a66b9ac6799659b5cc4"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "1ad5382887f6b230a0d99dab83d77653"
server_key_iv: "e8952ccc6269d40282a393b38be8a73d"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.457055 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:04.469076 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:04.470934 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:04.471102 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:04.471342 21394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:04.471657 21394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:04.471711 21394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.471747 21394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:04.471773 21394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.477825 21394 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:41969
I20260927 03:18:04.477883 21522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:41969 every 8 connection(s)
I20260927 03:18:04.478192 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:04.480556 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21394
I20260927 03:18:04.480722 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:04.480943 17618 external_mini_cluster.cc:1428] Setting key 30ff1202addc981a8af3b781a9fd5c79
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:04.489015 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.478892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:42845 (local address 127.17.52.190:42897)
0927 03:18:04.478936 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.478938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.479118 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:18:04.480152 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.480153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.480163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.480228 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.481018 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.481767 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.482939 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.483154 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.483213 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.485908 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.485910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.485910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.485916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.487355 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.487879 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.487879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.487880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.487936 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.488258 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.488259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.488259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.488385 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.488451 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.488863 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.488939 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:18:04.489289 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.478846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42897 (local address 127.17.52.130:42845)
0927 03:18:04.479006 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:04.479049 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:04.480050 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:04.480311 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:04.480314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:04.480602 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:04.480917 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.480922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.481875 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.481876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.482711 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.482713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.483239 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.483240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.483411 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.483952 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:04.483962 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:04.485804 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:04.487447 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.487455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.487458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.487800 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.488004 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.488005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.488005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.488160 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.488461 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:04.488467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:04.488679 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 03:18:04.488820 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 03:18:04.489016 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
I20260927 03:18:04.489938 21523 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42897
I20260927 03:18:04.490070 21523 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:04.490280 21523 heartbeater.cc:511] Master 127.17.52.190:42897 requested a full tablet report, sending...
I20260927 03:18:04.490690 21196 ts_manager.cc:194] Registered new tserver with Master: 6a806483ba354a66b9ac6799659b5cc4 (127.17.52.130:41969)
I20260927 03:18:04.491132 21196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:42845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:04.505497 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:04.587093 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:04.587337 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:04.587375 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:04.588518 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:04.588583 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:04.588689 21528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:04.590299 21528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:04.590569 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:04.590806 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:04.595753 21533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.595775 21536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:04.595749 21534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:04.595927 21528 server_base.cc:1034] running on GCE node
I20260927 03:18:04.596248 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:04.596554 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:04.597914 21528 hybrid_clock.cc:648] HybridClock initialized: now 1790479084597922 us; error 99 us; skew 500 ppm
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:04.600688 21528 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:04.601163 21528 webserver.cc:460] Webserver started at http://127.17.52.131:42105/ using document root <none> and password file <none>
I20260927 03:18:04.601359 21528 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:04.601403 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:04.601496 21528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:04.602501 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "0054ec51c2114220b61362832faceb12"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "f965c7740dc877f1dce0699851f4fd26"
server_key_iv: "ed49b27ea6c5f71bb8f8977642bb6577"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.602825 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "0054ec51c2114220b61362832faceb12"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "f965c7740dc877f1dce0699851f4fd26"
server_key_iv: "ed49b27ea6c5f71bb8f8977642bb6577"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.604235 21528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:04.605013 21544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.605216 21528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:18:04.605309 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "0054ec51c2114220b61362832faceb12"
format_stamp: "Formatted at 2026-09-27 03:18:04 on dist-test-slave-7j43"
server_key: "f965c7740dc877f1dce0699851f4fd26"
server_key_iv: "ed49b27ea6c5f71bb8f8977642bb6577"
server_key_version: "encryptionkey@0"
I20260927 03:18:04.605374 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:04.650238 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:04.652172 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:04.652289 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:04.652485 21528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:04.652871 21528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:04.652910 21528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.652961 21528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:04.652989 21528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:04.659121 21528 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:39569
I20260927 03:18:04.659216 21658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:39569 every 8 connection(s)
I20260927 03:18:04.659540 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:04.661006 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21528
I20260927 03:18:04.661098 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:04.661288 17618 external_mini_cluster.cc:1428] Setting key d34fed5e27e25ddbf6ca43b27bded70c
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:04.669276 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.660372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:58283 (local address 127.17.52.190:42897)
0927 03:18:04.660458 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.660460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.660664 (+ 204us) server_negotiation.cc:407] Connection header received
0927 03:18:04.662006 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.662007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.662017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.662057 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.662792 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.663372 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.664402 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.664667 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.664726 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.666838 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.666839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.666840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.666846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.668074 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.668428 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.668429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.668429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.668480 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.668744 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.668745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.668745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.668850 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.668906 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.669137 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.669206 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:18:04.669435 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.660334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42897 (local address 127.17.52.131:58283)
0927 03:18:04.660512 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:04.660549 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:04.661895 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:04.662137 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:04.662141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:04.662470 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:04.662670 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.662675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.663477 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.663478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.664322 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:04.664324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.664757 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.664758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:04.664865 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.665415 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:04.665423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:04.666753 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:04.668146 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.668151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.668153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.668361 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.668536 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:04.668537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:04.668537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:04.668685 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:04.668906 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:04.668910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:04.669023 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 03:18:04.669105 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:18:04.669220 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260927 03:18:04.669876 21659 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42897
I20260927 03:18:04.669965 21659 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:04.670099 21659 heartbeater.cc:511] Master 127.17.52.190:42897 requested a full tablet report, sending...
I20260927 03:18:04.670454 21196 ts_manager.cc:194] Registered new tserver with Master: 0054ec51c2114220b61362832faceb12 (127.17.52.131:39569)
I20260927 03:18:04.670804 21196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:58283
I20260927 03:18:04.673105 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:18:04.696818 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.689337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.17.52.129:38281)
0927 03:18:04.689514 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.689519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.689530 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:04.689555 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.689557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.689575 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.689642 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.690178 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.690883 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.691594 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.691958 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.692030 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.694166 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.694173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.694173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.694179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.695756 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.696037 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.696039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.696040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.696091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.696227 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.696227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.696227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.696354 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.696456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.696516 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.696652 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:04.707448 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:04.698272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.17.52.190:42897)
0927 03:18:04.698316 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:04.698318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:04.698386 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:18:04.698456 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:04.698457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:04.698467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:04.698671 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:04.699312 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.700149 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.701355 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:04.701651 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:04.701723 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:04.704785 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:04.704786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:04.704786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:04.704793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:04.706498 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.706776 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.706777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.706777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.706822 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:04.706930 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:04.706930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:04.706930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:04.707118 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:04.707191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:04.707321 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:18:04.707381 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:18:04.707690 21196 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952
I20260927 03:18:04.708058 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21259
I20260927 03:18:04.712400 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21394
I20260927 03:18:04.717439 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21528
I20260927 03:18:04.721443 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21164
2026-09-27T03:18:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1866 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21672](info): set up 2 sockets
Sep 27 03:18:04 dist-test-slave-7j43 krb5kdc[21672](info): commencing operation
krb5kdc: starting...
W20260927 03:18:05.788995 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:05 dist-test-slave-7j43 krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:05.900681 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:36047
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:05.980751 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:05.980947 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:05.980970 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:05.982136 21685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:05.982184 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:05.982198 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:05.982211 21685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:05.982223 21685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:05.983758 21685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:05.984017 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:05.984244 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:05.989035 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:05.989053 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:05.989038 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:05.989434 21685 server_base.cc:1034] running on GCE node
I20260927 03:18:05.989691 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:05.990022 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:05.991165 21685 hybrid_clock.cc:648] HybridClock initialized: now 1790479085991155 us; error 27 us; skew 500 ppm
Sep 27 03:18:05 dist-test-slave-7j43 krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479085, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:05.993649 21685 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:05.994026 21685 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:05.994078 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:05.994169 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:05.995182 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "05fe836370604535b5f08cd583669169"
format_stamp: "Formatted at 2026-09-27 03:18:05 on dist-test-slave-7j43"
server_key: "336fa9b43f0317fcee3fb01663973a02"
server_key_iv: "ab08d0cad058e8aed2e54349c5a152bb"
server_key_version: "encryptionkey@0"
I20260927 03:18:05.995496 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "05fe836370604535b5f08cd583669169"
format_stamp: "Formatted at 2026-09-27 03:18:05 on dist-test-slave-7j43"
server_key: "336fa9b43f0317fcee3fb01663973a02"
server_key_iv: "ab08d0cad058e8aed2e54349c5a152bb"
server_key_version: "encryptionkey@0"
I20260927 03:18:05.996788 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:05.997479 21699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:05.997658 21685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:05.997752 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "05fe836370604535b5f08cd583669169"
format_stamp: "Formatted at 2026-09-27 03:18:05 on dist-test-slave-7j43"
server_key: "336fa9b43f0317fcee3fb01663973a02"
server_key_iv: "ab08d0cad058e8aed2e54349c5a152bb"
server_key_version: "encryptionkey@0"
I20260927 03:18:05.997854 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:06.014397 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:06.016247 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:06.016407 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:06.020394 21685 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:36047
I20260927 03:18:06.020437 21751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:36047 every 8 connection(s)
I20260927 03:18:06.020776 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:06.021723 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:06.024014 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: Bootstrap starting.
I20260927 03:18:06.024690 21752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:06.024986 21752 log.cc:824] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:06.025347 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21685
I20260927 03:18:06.025455 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:06.025664 17618 external_mini_cluster.cc:1428] Setting key 1945839e15293dd6c4159a3c49bd1028
I20260927 03:18:06.025753 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: No bootstrap required, opened a new log
I20260927 03:18:06.027557 21752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } }
I20260927 03:18:06.027722 21752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:06.027760 21752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fe836370604535b5f08cd583669169, State: Initialized, Role: FOLLOWER
I20260927 03:18:06.027930 21752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } }
I20260927 03:18:06.028019 21752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:06.028056 21752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:06.028117 21752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:06.029083 21752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } }
I20260927 03:18:06.029178 21752 leader_election.cc:302] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05fe836370604535b5f08cd583669169; no voters:
I20260927 03:18:06.029358 21752 leader_election.cc:288] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:06.029408 21757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:06.029578 21757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [term 1 LEADER]: Becoming Leader. State: Replica: 05fe836370604535b5f08cd583669169, State: Running, Role: LEADER
I20260927 03:18:06.029640 21752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:06.029752 21757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } }
I20260927 03:18:06.030189 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fe836370604535b5f08cd583669169. Latest consensus state: current_term: 1 leader_uuid: "05fe836370604535b5f08cd583669169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } } }
I20260927 03:18:06.030284 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:06.030463 21766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:06.030432 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05fe836370604535b5f08cd583669169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fe836370604535b5f08cd583669169" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 36047 } } }
I20260927 03:18:06.030514 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:06.030951 21766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:06.032519 21766 catalog_manager.cc:1347] Generated new cluster ID: 87c5af80608f44f7881525eb57b17ac9
I20260927 03:18:06.032589 21766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:06.037262 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.026186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.17.52.190:36047)
0927 03:18:06.026366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.026381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.026405 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:18:06.027539 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.027545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.027601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.027711 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.028501 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.029675 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.030788 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.031410 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.031515 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.033993 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.033996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.033996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:06.034008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.035663 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.036039 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.036042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.036042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.036147 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.036364 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.036365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.036366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.036668 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.036846 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.036912 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.037046 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260927 03:18:06.056548 21766 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:06.057301 21766 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:06.062538 21766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: Generated new TSK 0
I20260927 03:18:06.062707 21766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:06.095679 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:06.201654 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:06.201869 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:06.201894 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:06.203083 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:06.203126 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:06.203154 21778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:06.204774 21778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:06.205029 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:06.205273 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:06.210363 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.210471 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.210489 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:06.210575 21778 server_base.cc:1034] running on GCE node
I20260927 03:18:06.210902 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:06.211196 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:06.212370 21778 hybrid_clock.cc:648] HybridClock initialized: now 1790479086212347 us; error 48 us; skew 500 ppm
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:06.214952 21778 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:06.215263 21778 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:06.215308 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:06.215399 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:06.216418 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "bf838c98aa924f30a73053b986b15f29"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "743e6250825fa55dacdd7b6b9118006b"
server_key_iv: "cd8a14cd5fac66fd49aa652807c0a712"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.216800 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "bf838c98aa924f30a73053b986b15f29"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "743e6250825fa55dacdd7b6b9118006b"
server_key_iv: "cd8a14cd5fac66fd49aa652807c0a712"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.218055 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:06.218748 21792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.218930 21778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:06.219017 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "bf838c98aa924f30a73053b986b15f29"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "743e6250825fa55dacdd7b6b9118006b"
server_key_iv: "cd8a14cd5fac66fd49aa652807c0a712"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.219084 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:06.255203 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:06.257365 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:06.257522 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:06.257789 21778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:06.258133 21778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:06.258189 21778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.258224 21778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:06.258252 21778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.264230 21778 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:39801
I20260927 03:18:06.264262 21906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:39801 every 8 connection(s)
I20260927 03:18:06.264544 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:06.271914 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21778
I20260927 03:18:06.272056 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:06.272265 17618 external_mini_cluster.cc:1428] Setting key 5e14487aa8758f7786f75141bb322a41
I20260927 03:18:06.274212 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.265316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:44877 (local address 127.17.52.190:36047)
0927 03:18:06.265402 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.265405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.265576 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:18:06.266116 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.266117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.266133 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.266181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.267160 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.267660 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.268655 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.268870 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.268923 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.271044 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.271045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.271046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:06.271053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.272532 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.273262 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.273263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.273263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.273318 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.273599 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.273599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.273600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.273699 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.273748 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.274075 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.274148 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:18:06.274534 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.265270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:36047 (local address 127.17.52.129:44877)
0927 03:18:06.265458 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.265494 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.266002 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.266272 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.266275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.266529 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:06.267051 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.267059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.267741 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.267742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.268549 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.268551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.268943 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.268944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.269045 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.269600 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:06.269608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:06.270958 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:06.272863 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.272870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.272873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.273188 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.273384 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.273385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.273385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.273537 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.273758 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:06.273765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:06.273939 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.274056 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.274237 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260927 03:18:06.275071 21907 heartbeater.cc:347] Connected to a master server at 127.17.52.190:36047
I20260927 03:18:06.275184 21907 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:06.275343 21907 heartbeater.cc:511] Master 127.17.52.190:36047 requested a full tablet report, sending...
I20260927 03:18:06.275906 21716 ts_manager.cc:194] Registered new tserver with Master: bf838c98aa924f30a73053b986b15f29 (127.17.52.129:39801)
I20260927 03:18:06.276507 21716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:44877
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:06.297109 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:06.377702 21912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:06.377920 21912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:06.377944 21912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:06.379096 21912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:06.379137 21912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:06.379192 21912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:06.380705 21912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:06.380960 21912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:06.381206 21912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:06.386160 21918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.386166 21917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.386166 21920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:06.386292 21912 server_base.cc:1034] running on GCE node
I20260927 03:18:06.386546 21912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:06.386819 21912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:06.387966 21912 hybrid_clock.cc:648] HybridClock initialized: now 1790479086387948 us; error 35 us; skew 500 ppm
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:06.390383 21912 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:06.390712 21912 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:06.390810 21912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:06.390946 21912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:06.392002 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "163c47fc125345619e81376ead16994d"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "cff7b71a169db8a7830b88ade6522163"
server_key_iv: "e522ebb9f286ea41a744316a232b9e1d"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.392359 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "163c47fc125345619e81376ead16994d"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "cff7b71a169db8a7830b88ade6522163"
server_key_iv: "e522ebb9f286ea41a744316a232b9e1d"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.393659 21912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:06.394399 21926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.394548 21912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:06.394646 21912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "163c47fc125345619e81376ead16994d"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "cff7b71a169db8a7830b88ade6522163"
server_key_iv: "e522ebb9f286ea41a744316a232b9e1d"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.394721 21912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:06.416038 21912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:06.418179 21912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:06.418347 21912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:06.418586 21912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:06.418912 21912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:06.418974 21912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.419015 21912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:06.419044 21912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.425045 21912 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:33543
I20260927 03:18:06.425077 22040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:33543 every 8 connection(s)
I20260927 03:18:06.425364 21912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:06.432104 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 21912
I20260927 03:18:06.432228 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:06.432438 17618 external_mini_cluster.cc:1428] Setting key e5dd9d303cb7928da921a287cc780b49
I20260927 03:18:06.434885 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.426059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:45243 (local address 127.17.52.190:36047)
0927 03:18:06.426132 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.426134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.426338 (+ 204us) server_negotiation.cc:407] Connection header received
0927 03:18:06.426851 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.426853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.426865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.426921 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.427865 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.428370 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.429548 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.429739 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.429828 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.431944 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.431945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.431946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:06.431953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.433484 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.433886 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.433887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.433888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.433941 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.434195 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.434196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.434196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.434349 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.434414 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.434751 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.434822 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:18:06.435133 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.426044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:36047 (local address 127.17.52.130:45243)
0927 03:18:06.426209 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.426252 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.426744 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.426994 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.426998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.427260 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:06.427765 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.427772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.428461 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.428462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.429450 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.429452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.429835 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.429836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.430051 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.430485 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:06.430493 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:06.431854 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:06.433561 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.433567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.433569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.433796 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.434012 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.434013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.434013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.434119 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.434442 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:06.434447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:06.434605 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.434717 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.434880 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260927 03:18:06.435627 22041 heartbeater.cc:347] Connected to a master server at 127.17.52.190:36047
I20260927 03:18:06.435724 22041 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:06.435876 22041 heartbeater.cc:511] Master 127.17.52.190:36047 requested a full tablet report, sending...
I20260927 03:18:06.436204 21716 ts_manager.cc:194] Registered new tserver with Master: 163c47fc125345619e81376ead16994d (127.17.52.130:33543)
I20260927 03:18:06.436630 21716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:45243
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:06.455662 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:06.536283 22046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:06.536509 22046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:06.536550 22046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:06.537708 22046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:06.537772 22046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:06.537817 22046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:06.539305 22046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:36047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:06.539554 22046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:06.539769 22046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:06.544517 22054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.544543 22051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.544720 22052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:06.544793 22046 server_base.cc:1034] running on GCE node
I20260927 03:18:06.544989 22046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:06.545172 22046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:06.546309 22046 hybrid_clock.cc:648] HybridClock initialized: now 1790479086546289 us; error 34 us; skew 500 ppm
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:06.548707 22046 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:06.549104 22046 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:06.549152 22046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:06.549299 22046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:06.550328 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "682262210149445eb7c3260c66d2515a"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "92e3bbdaff45dafd02d717238486b889"
server_key_iv: "420d57eb64c9d7af778485f84ef3d80e"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.550683 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "682262210149445eb7c3260c66d2515a"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "92e3bbdaff45dafd02d717238486b889"
server_key_iv: "420d57eb64c9d7af778485f84ef3d80e"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.551954 22046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:06.552780 22060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.552966 22046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:06.553038 22046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "682262210149445eb7c3260c66d2515a"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "92e3bbdaff45dafd02d717238486b889"
server_key_iv: "420d57eb64c9d7af778485f84ef3d80e"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.553138 22046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:06.563131 22046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:06.565056 22046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:06.565224 22046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:06.565467 22046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:06.565832 22046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:06.565872 22046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.565918 22046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:06.565950 22046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.572291 22046 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42523
I20260927 03:18:06.572322 22174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42523 every 8 connection(s)
I20260927 03:18:06.572686 22046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:06 dist-test-slave-7j43 krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479086, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:06.580471 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22046
I20260927 03:18:06.580585 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:06.580785 17618 external_mini_cluster.cc:1428] Setting key b8c991f0d56ff0d728fd3d09aeac92a3
I20260927 03:18:06.581700 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.573384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:49197 (local address 127.17.52.190:36047)
0927 03:18:06.573454 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.573456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.573701 (+ 245us) server_negotiation.cc:407] Connection header received
0927 03:18:06.574175 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.574176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.574186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.574232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.575133 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.575618 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.576691 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.576887 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.576938 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.578941 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.578942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.578943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:06.578949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.580122 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.580620 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.580621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.580622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.580688 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.580990 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.580991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.580991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.581128 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.581198 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.581523 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.581608 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:18:06.581878 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.573372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:36047 (local address 127.17.52.131:49197)
0927 03:18:06.573564 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.573612 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.574068 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.574294 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.574297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.574555 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:06.575047 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.575053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.575715 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.575716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.576532 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.576534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.576961 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.576962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.577068 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.577510 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:06.577518 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:06.578849 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:06.580194 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.580200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.580201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.580510 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.580757 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:06.580758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:06.580758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:06.580902 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:06.581201 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:06.581207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:06.581374 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.581478 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.581663 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260927 03:18:06.582386 22175 heartbeater.cc:347] Connected to a master server at 127.17.52.190:36047
I20260927 03:18:06.582497 22175 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:06.582647 22175 heartbeater.cc:511] Master 127.17.52.190:36047 requested a full tablet report, sending...
I20260927 03:18:06.583022 21716 ts_manager.cc:194] Registered new tserver with Master: 682262210149445eb7c3260c66d2515a (127.17.52.131:42523)
I20260927 03:18:06.583408 21716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:49197
I20260927 03:18:06.592283 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:18:06.599062 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.593589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.17.52.190:36047)
0927 03:18:06.593650 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.593653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.593714 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:18:06.593770 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.593771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.593779 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.593826 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.594319 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.594798 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.595476 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.595645 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.595692 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.596866 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.596867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.596867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:06.596873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.598092 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.598384 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.598385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.598385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.598431 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:06.598643 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:06.598644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:06.598644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:06.598786 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.598861 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.598948 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.599002 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:18:06.600835 21716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:06.602324 21716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:06.608798 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42523 (local address 127.0.0.1:42850)
0927 03:18:06.604711 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.604771 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.604861 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.605078 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.605081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.605083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:06.605161 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.605163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606572 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.606573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.607647 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.607649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608477 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.608534 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608547 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.608618 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.608686 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260927 03:18:06.608958 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.17.52.131:42523)
0927 03:18:06.604884 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.604888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.604900 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:18:06.604945 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.604947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.604960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.605062 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:06.605341 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606461 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.607836 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608364 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608734 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608795 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.608822 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.608869 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260927 03:18:06.608970 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.17.52.130:33543)
0927 03:18:06.604887 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.604891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.605191 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:18:06.605236 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.605238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.605253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.605328 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:06.605519 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606194 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.607704 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608483 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608546 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608631 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.608715 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.608754 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260927 03:18:06.608896 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33543 (local address 127.0.0.1:51328)
0927 03:18:06.605092 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.605100 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.605138 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.605415 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.605416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.605417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:06.605455 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.605456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606309 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.606309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.607628 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.607630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608548 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.608638 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608643 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.608701 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.608745 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260927 03:18:06.609148 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.17.52.129:39801)
0927 03:18:06.604808 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.604814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.605034 (+ 220us) server_negotiation.cc:407] Connection header received
0927 03:18:06.605079 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.605081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.605095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.605165 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:06.605401 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606479 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.607624 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608351 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608429 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608489 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.609024 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.609073 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260927 03:18:06.609320 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.604429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:39801 (local address 127.0.0.1:45384)
0927 03:18:06.604956 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.604967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.605010 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.605255 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.605256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.605259 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:06.605316 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.605318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.606585 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.606586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.607539 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.607541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.608446 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.608447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.608532 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.608547 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.608637 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.608686 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":10,"thread_start_us":79,"threads_started":1}
I20260927 03:18:06.609704 22108 tablet_service.cc:1467] Processing CreateTablet for tablet 63a676ede1a14273abe2cc4486d04bde (DEFAULT_TABLE table=test-table [id=cc74d21b7d9d4fc49dc20eb4d514db7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:06.609975 21975 tablet_service.cc:1467] Processing CreateTablet for tablet 63a676ede1a14273abe2cc4486d04bde (DEFAULT_TABLE table=test-table [id=cc74d21b7d9d4fc49dc20eb4d514db7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:06.610102 22108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a676ede1a14273abe2cc4486d04bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:06.610203 21841 tablet_service.cc:1467] Processing CreateTablet for tablet 63a676ede1a14273abe2cc4486d04bde (DEFAULT_TABLE table=test-table [id=cc74d21b7d9d4fc49dc20eb4d514db7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:06.610327 21975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a676ede1a14273abe2cc4486d04bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:06.610489 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a676ede1a14273abe2cc4486d04bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:06.613036 22192 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Bootstrap starting.
I20260927 03:18:06.613739 22193 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Bootstrap starting.
I20260927 03:18:06.613948 22192 tablet_bootstrap.cc:654] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:06.614459 22193 tablet_bootstrap.cc:654] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:06.614670 22192 log.cc:824] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:06.615085 22193 log.cc:824] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:06.615103 22194 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Bootstrap starting.
I20260927 03:18:06.615731 22194 tablet_bootstrap.cc:654] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:06.615863 22193 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: No bootstrap required, opened a new log
I20260927 03:18:06.615939 22193 ts_tablet_manager.cc:1397] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:18:06.616554 22192 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: No bootstrap required, opened a new log
I20260927 03:18:06.616649 22192 ts_tablet_manager.cc:1397] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 03:18:06.616948 22194 log.cc:824] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:06.617626 22194 tablet_bootstrap.cc:492] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: No bootstrap required, opened a new log
I20260927 03:18:06.617707 22194 ts_tablet_manager.cc:1397] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:18:06.618016 22193 raft_consensus.cc:348] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.618192 22193 raft_consensus.cc:374] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:06.618229 22193 raft_consensus.cc:729] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163c47fc125345619e81376ead16994d, State: Initialized, Role: FOLLOWER
I20260927 03:18:06.618387 22193 consensus_queue.cc:260] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.618669 22193 ts_tablet_manager.cc:1428] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:18:06.618624 22192 raft_consensus.cc:348] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.618784 22192 raft_consensus.cc:374] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:06.618798 22041 heartbeater.cc:503] Master 127.17.52.190:36047 was elected leader, sending a full tablet report...
I20260927 03:18:06.618830 22192 raft_consensus.cc:729] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682262210149445eb7c3260c66d2515a, State: Initialized, Role: FOLLOWER
I20260927 03:18:06.618938 22192 consensus_queue.cc:260] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.619257 22175 heartbeater.cc:503] Master 127.17.52.190:36047 was elected leader, sending a full tablet report...
I20260927 03:18:06.619583 22194 raft_consensus.cc:348] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.619606 22192 ts_tablet_manager.cc:1428] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:18:06.619736 22194 raft_consensus.cc:374] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:06.619773 22194 raft_consensus.cc:729] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf838c98aa924f30a73053b986b15f29, State: Initialized, Role: FOLLOWER
I20260927 03:18:06.619874 22194 consensus_queue.cc:260] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.620165 22194 ts_tablet_manager.cc:1428] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:18:06.620285 21907 heartbeater.cc:503] Master 127.17.52.190:36047 was elected leader, sending a full tablet report...
W20260927 03:18:06.675945 22042 tablet.cc:2369] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:18:06.758642 22200 raft_consensus.cc:482] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:06.758767 22200 raft_consensus.cc:504] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.759097 22200 leader_election.cc:288] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 682262210149445eb7c3260c66d2515a (127.17.52.131:42523), 163c47fc125345619e81376ead16994d (127.17.52.130:33543)
I20260927 03:18:06.761910 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.759346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59125 (local address 127.17.52.130:33543)
0927 03:18:06.759408 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.759410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.759439 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:18:06.759467 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.759470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.759484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.759550 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:06.759694 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.760173 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761181 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.761555 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.761659 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.761810 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.761844 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:18:06.761910 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.759268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33543 (local address 127.17.52.129:59125)
0927 03:18:06.759328 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.759341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.759387 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.759556 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.759557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.759560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:06.759611 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.759613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.760252 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.760253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.761090 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.761093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.761649 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.761719 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.761722 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.761802 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.761840 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:18:06.762018 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.759349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:53221 (local address 127.17.52.131:42523)
0927 03:18:06.759423 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.759425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.759596 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:18:06.759662 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.759663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.759672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.759739 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:06.759897 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.760374 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761292 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.761661 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761743 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.761751 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.761913 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.761948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 03:18:06.762208 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.759275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42523 (local address 127.17.52.129:53221)
0927 03:18:06.759522 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.759530 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:06.759600 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:06.759742 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:06.759742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:06.759744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:06.759820 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.759821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.760458 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.760459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.761199 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:06.761201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.761742 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.761743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:06.761824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.761827 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:06.761896 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:18:06.761933 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":118,"thread_start_us":77,"threads_started":1}
I20260927 03:18:06.762253 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a676ede1a14273abe2cc4486d04bde" candidate_uuid: "bf838c98aa924f30a73053b986b15f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163c47fc125345619e81376ead16994d" is_pre_election: true
I20260927 03:18:06.762394 21995 raft_consensus.cc:2454] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf838c98aa924f30a73053b986b15f29 in term 0.
I20260927 03:18:06.762574 22128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a676ede1a14273abe2cc4486d04bde" candidate_uuid: "bf838c98aa924f30a73053b986b15f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682262210149445eb7c3260c66d2515a" is_pre_election: true
I20260927 03:18:06.762674 21795 leader_election.cc:302] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 163c47fc125345619e81376ead16994d, bf838c98aa924f30a73053b986b15f29; no voters:
I20260927 03:18:06.762717 22128 raft_consensus.cc:2454] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf838c98aa924f30a73053b986b15f29 in term 0.
I20260927 03:18:06.762799 22200 raft_consensus.cc:2781] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:06.762876 22200 raft_consensus.cc:482] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:06.762912 22200 raft_consensus.cc:3037] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:06.763552 22200 raft_consensus.cc:504] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
I20260927 03:18:06.763695 22200 leader_election.cc:288] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [CANDIDATE]: Term 1 election: Requested vote from peers 682262210149445eb7c3260c66d2515a (127.17.52.131:42523), 163c47fc125345619e81376ead16994d (127.17.52.130:33543)
I20260927 03:18:06.763836 22128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a676ede1a14273abe2cc4486d04bde" candidate_uuid: "bf838c98aa924f30a73053b986b15f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682262210149445eb7c3260c66d2515a"
I20260927 03:18:06.763906 22128 raft_consensus.cc:3037] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:06.764600 22128 raft_consensus.cc:2454] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf838c98aa924f30a73053b986b15f29 in term 1.
I20260927 03:18:06.764730 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a676ede1a14273abe2cc4486d04bde" candidate_uuid: "bf838c98aa924f30a73053b986b15f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163c47fc125345619e81376ead16994d"
I20260927 03:18:06.764779 21796 leader_election.cc:302] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682262210149445eb7c3260c66d2515a, bf838c98aa924f30a73053b986b15f29; no voters:
I20260927 03:18:06.764807 21995 raft_consensus.cc:3037] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:06.764869 22200 raft_consensus.cc:2781] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:06.765059 22200 raft_consensus.cc:686] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [term 1 LEADER]: Becoming Leader. State: Replica: bf838c98aa924f30a73053b986b15f29, State: Running, Role: LEADER
I20260927 03:18:06.765165 22200 consensus_queue.cc:237] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } }
W20260927 03:18:06.765209 21908 tablet.cc:2369] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:18:06.765688 21995 raft_consensus.cc:2454] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf838c98aa924f30a73053b986b15f29 in term 1.
I20260927 03:18:06.766294 21716 catalog_manager.cc:5495] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf838c98aa924f30a73053b986b15f29 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "bf838c98aa924f30a73053b986b15f29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf838c98aa924f30a73053b986b15f29" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 39801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:06.789752 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.787486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.17.52.129:39801)
0927 03:18:06.787555 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.787557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.787566 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:06.787612 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.787613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.787630 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.787691 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:06.787824 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.788290 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.789052 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:06.789228 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:06.789290 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:06.789421 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:06.789514 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:06.789573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.789606 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.789622 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:18:06.791378 21995 raft_consensus.cc:1261] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 1 FOLLOWER]: Refusing update from remote peer bf838c98aa924f30a73053b986b15f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:06.791420 22128 raft_consensus.cc:1261] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 1 FOLLOWER]: Refusing update from remote peer bf838c98aa924f30a73053b986b15f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:06.791693 22200 consensus_queue.cc:1035] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163c47fc125345619e81376ead16994d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:06.791875 22202 consensus_queue.cc:1035] T 63a676ede1a14273abe2cc4486d04bde P bf838c98aa924f30a73053b986b15f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682262210149445eb7c3260c66d2515a" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:06.793998 22205 mvcc.cc:205] Tried to move back new op lower bound from 7333802339495124992 to 7333802339391569920. Current Snapshot: MvccSnapshot[applied={T|T < 7333802339495124992}]
I20260927 03:18:06.794263 22207 mvcc.cc:205] Tried to move back new op lower bound from 7333802339495124992 to 7333802339391569920. Current Snapshot: MvccSnapshot[applied={T|T < 7333802339495124992}]
I20260927 03:18:06.796191 21715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:06.796298 21715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:06.797034 21715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05fe836370604535b5f08cd583669169: Sending DeleteTablet for 3 replicas of tablet 63a676ede1a14273abe2cc4486d04bde
I20260927 03:18:06.797361 21975 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a676ede1a14273abe2cc4486d04bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:06 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:51328
I20260927 03:18:06.797578 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21778
I20260927 03:18:06.797588 22214 tablet_replica.cc:330] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: stopping tablet replica
I20260927 03:18:06.797586 22108 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a676ede1a14273abe2cc4486d04bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:06 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:42850
I20260927 03:18:06.797677 22214 raft_consensus.cc:2229] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:06.797770 22214 raft_consensus.cc:2258] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:06.798460 22215 tablet_replica.cc:330] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: stopping tablet replica
I20260927 03:18:06.798569 22215 raft_consensus.cc:2229] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:06.798664 22215 raft_consensus.cc:2258] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:06.799031 22215 ts_tablet_manager.cc:1905] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:06.800467 22215 ts_tablet_manager.cc:1918] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:06.800540 22215 log.cc:1196] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/63a676ede1a14273abe2cc4486d04bde
I20260927 03:18:06.800871 22215 ts_tablet_manager.cc:1939] T 63a676ede1a14273abe2cc4486d04bde P 682262210149445eb7c3260c66d2515a: Deleting consensus metadata
I20260927 03:18:06.801295 21703 catalog_manager.cc:4841] TS 682262210149445eb7c3260c66d2515a (127.17.52.131:42523): tablet 63a676ede1a14273abe2cc4486d04bde (table test-table [id=cc74d21b7d9d4fc49dc20eb4d514db7e]) successfully deleted
W20260927 03:18:06.803733 21700 connection.cc:535] client connection to 127.17.52.129:39801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:18:06.803857 21700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:18:06.804014 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21912
I20260927 03:18:06.804033 22214 ts_tablet_manager.cc:1905] T 63a676ede1a14273abe2cc4486d04bde P 163c47fc125345619e81376ead16994d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:06.804796 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.804244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:39801 (local address 127.0.0.1:45394)
0927 03:18:06.804714 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.804761 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
W20260927 03:18:06.804947 21700 catalog_manager.cc:4584] TS bf838c98aa924f30a73053b986b15f29 (127.17.52.129:39801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63a676ede1a14273abe2cc4486d04bde: Network error: Client connection negotiation failed: client connection to 127.17.52.129:39801: connect: Connection refused (error 111)
I20260927 03:18:06.809273 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22046
I20260927 03:18:06.809288 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.809177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33543 (local address 127.0.0.1:51344)
0927 03:18:06.809230 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:06.809262 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.130:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:18:06.813616 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 21685
2026-09-27T03:18:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2092 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T03:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:06Z Disabled control of system clock
I20260927 03:18:06.840893 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42697
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:18:06.921634 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:06.921840 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:06.921861 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:06.922919 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:06.922953 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:06.922966 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:06.922978 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:06.922991 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:06.924235 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:42697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:06.924494 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:06.924765 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:06.929535 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.929553 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:06.929535 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:06.929814 22220 server_base.cc:1034] running on GCE node
I20260927 03:18:06.930066 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:06.930415 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:06.931592 22220 hybrid_clock.cc:648] HybridClock initialized: now 1790479086931568 us; error 32 us; skew 500 ppm
I20260927 03:18:06.933030 22220 webserver.cc:460] Webserver started at http://127.17.52.190:42937/ using document root <none> and password file <none>
I20260927 03:18:06.933318 22220 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:06.933393 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:06.933559 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:06.934671 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "05182c9a14a94acc961a6dd2b0020df3"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "493956248e5aeab9508c45f28bb25b8d"
server_key_iv: "fbf24610ffc66348c4d8f157b6b0abe8"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.935056 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "05182c9a14a94acc961a6dd2b0020df3"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "493956248e5aeab9508c45f28bb25b8d"
server_key_iv: "fbf24610ffc66348c4d8f157b6b0abe8"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.936388 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:06.937237 22234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:06.937423 22220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:06.937525 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "05182c9a14a94acc961a6dd2b0020df3"
format_stamp: "Formatted at 2026-09-27 03:18:06 on dist-test-slave-7j43"
server_key: "493956248e5aeab9508c45f28bb25b8d"
server_key_iv: "fbf24610ffc66348c4d8f157b6b0abe8"
server_key_version: "encryptionkey@0"
I20260927 03:18:06.937616 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:06.940915 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:06.941171 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:06.941316 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:06.945066 22220 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:42697
I20260927 03:18:06.945096 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:42697 every 8 connection(s)
I20260927 03:18:06.945425 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:06.946162 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:06.948968 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: Bootstrap starting.
I20260927 03:18:06.949507 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:06.949740 22288 log.cc:824] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:06.950364 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: No bootstrap required, opened a new log
I20260927 03:18:06.951866 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } }
I20260927 03:18:06.951998 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:06.952020 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05182c9a14a94acc961a6dd2b0020df3, State: Initialized, Role: FOLLOWER
I20260927 03:18:06.952165 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } }
I20260927 03:18:06.952243 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:06.952266 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:06.952294 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:06.952855 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } }
I20260927 03:18:06.952939 22288 leader_election.cc:302] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05182c9a14a94acc961a6dd2b0020df3; no voters:
I20260927 03:18:06.953053 22288 leader_election.cc:288] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:06.953109 22291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:06.953287 22291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [term 1 LEADER]: Becoming Leader. State: Replica: 05182c9a14a94acc961a6dd2b0020df3, State: Running, Role: LEADER
I20260927 03:18:06.953358 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:06.953424 22291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } }
I20260927 03:18:06.953835 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05182c9a14a94acc961a6dd2b0020df3. Latest consensus state: current_term: 1 leader_uuid: "05182c9a14a94acc961a6dd2b0020df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } } }
I20260927 03:18:06.953933 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:06.954035 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05182c9a14a94acc961a6dd2b0020df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05182c9a14a94acc961a6dd2b0020df3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42697 } } }
I20260927 03:18:06.954142 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:06.954144 22300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:06.954733 22300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:06.954892 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22220
I20260927 03:18:06.954991 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:06.955166 17618 external_mini_cluster.cc:1428] Setting key 63137c0ea470c0937aa66fd8a19871a7
I20260927 03:18:06.956468 22300 catalog_manager.cc:1347] Generated new cluster ID: a4d87657aa334335a69c0d6e1b81bd6c
I20260927 03:18:06.956522 22300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:06.963791 22300 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:06.963930 22300 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:06.964802 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:06.955600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.17.52.190:42697)
0927 03:18:06.956211 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:06.956222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:06.956253 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:18:06.956368 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:06.956375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:06.956417 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:06.963544 (+ 7127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:06.964314 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:06.964317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:06.964318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:06.964322 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:06.964366 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:06.964380 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:06.964443 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:06.964534 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:18:06.964609 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":77,"threads_started":1}
I20260927 03:18:06.969266 22300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: Generated new TSK 0
I20260927 03:18:06.969457 22300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:06.970860 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--builtin_ntp_servers=127.17.52.148:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:18:07.052037 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:07.052246 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:07.052269 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:07.053608 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:07.053653 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:07.053680 22312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:07.055228 22312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:07.055452 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:07.055753 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:07.060561 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.060590 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.060851 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.060984 22312 server_base.cc:1034] running on GCE node
I20260927 03:18:07.061275 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:07.061546 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:07.062717 22312 hybrid_clock.cc:648] HybridClock initialized: now 1790479087062705 us; error 31 us; skew 500 ppm
I20260927 03:18:07.063939 22312 webserver.cc:460] Webserver started at http://127.17.52.129:37759/ using document root <none> and password file <none>
I20260927 03:18:07.064170 22312 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:07.064241 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:07.064399 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:07.065587 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "5d311ae7042e47b1b09dd6ebec4464b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "183b96e9f3c0ccc1449f0d4e0539d822"
server_key_iv: "cb58b23fcb30f247f688de49bd2b0d6a"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.065938 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "5d311ae7042e47b1b09dd6ebec4464b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "183b96e9f3c0ccc1449f0d4e0539d822"
server_key_iv: "cb58b23fcb30f247f688de49bd2b0d6a"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.067248 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:07.067977 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.068145 22312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:07.068236 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "5d311ae7042e47b1b09dd6ebec4464b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "183b96e9f3c0ccc1449f0d4e0539d822"
server_key_iv: "cb58b23fcb30f247f688de49bd2b0d6a"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.068324 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:07.071554 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:07.071805 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:07.071947 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:07.072148 22312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:07.072449 22312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:07.072505 22312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.072543 22312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:07.072584 22312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.078449 22312 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:33573
I20260927 03:18:07.078490 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:33573 every 8 connection(s)
I20260927 03:18:07.078855 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:07.082015 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.079594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:37571 (local address 127.17.52.190:42697)
0927 03:18:07.079648 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.079650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.079907 (+ 257us) server_negotiation.cc:407] Connection header received
0927 03:18:07.080460 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.080461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.080466 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.080506 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.081819 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.081820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.081821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.081824 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.081838 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.081842 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.081890 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.081950 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.081963 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:18:07.082213 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.079562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42697 (local address 127.17.52.129:37571)
0927 03:18:07.079764 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.079813 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.080361 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.080622 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.080626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.080636 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.081647 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.081648 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.081686 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.081720 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.081889 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.081892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.081894 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.081950 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.081997 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260927 03:18:07.082715 22441 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42697
I20260927 03:18:07.082789 22441 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:07.082938 22441 heartbeater.cc:511] Master 127.17.52.190:42697 requested a full tablet report, sending...
I20260927 03:18:07.083423 22251 ts_manager.cc:194] Registered new tserver with Master: 5d311ae7042e47b1b09dd6ebec4464b4 (127.17.52.129:33573)
I20260927 03:18:07.085593 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22312
I20260927 03:18:07.085680 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:07.085821 17618 external_mini_cluster.cc:1428] Setting key 3211bcc3d9eae6eb6eb527642f13f208
I20260927 03:18:07.086633 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--builtin_ntp_servers=127.17.52.148:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:18:07.166217 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:07.166422 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:07.166445 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:07.167641 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:07.167680 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:07.167707 22444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:07.169009 22444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:07.169199 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:07.169458 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:07.174343 22452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.174340 22449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.174376 22450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.174623 22444 server_base.cc:1034] running on GCE node
I20260927 03:18:07.174892 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:07.175122 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:07.176312 22444 hybrid_clock.cc:648] HybridClock initialized: now 1790479087176297 us; error 35 us; skew 500 ppm
I20260927 03:18:07.177750 22444 webserver.cc:460] Webserver started at http://127.17.52.130:37759/ using document root <none> and password file <none>
I20260927 03:18:07.178023 22444 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:07.178094 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:07.178227 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:07.179253 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "5fe578df15b649b98d8b740bac7c66e1"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "cd30fc3d41c21bf517af5b8eafde12bd"
server_key_iv: "466dc35ebdd101a7672cc2df46960344"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.179617 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "5fe578df15b649b98d8b740bac7c66e1"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "cd30fc3d41c21bf517af5b8eafde12bd"
server_key_iv: "466dc35ebdd101a7672cc2df46960344"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.180955 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:18:07.181700 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.181883 22444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:07.181995 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "5fe578df15b649b98d8b740bac7c66e1"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "cd30fc3d41c21bf517af5b8eafde12bd"
server_key_iv: "466dc35ebdd101a7672cc2df46960344"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.182104 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:07.185650 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:07.185920 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:07.186067 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:07.186272 22444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:07.186602 22444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:07.186658 22444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.186702 22444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:07.186733 22444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.193115 22444 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:36245
I20260927 03:18:07.193149 22572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:36245 every 8 connection(s)
I20260927 03:18:07.193490 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:07.196373 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.194263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:59219 (local address 127.17.52.190:42697)
0927 03:18:07.194324 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.194325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.194561 (+ 236us) server_negotiation.cc:407] Connection header received
0927 03:18:07.195073 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.195075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.195081 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.195139 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.196183 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.196184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.196184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.196188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.196202 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.196205 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.196250 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.196309 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.196321 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:18:07.196566 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.194247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42697 (local address 127.17.52.130:59219)
0927 03:18:07.194445 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.194476 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.194897 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.195206 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.195209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.195217 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.196052 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.196053 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.196079 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.196105 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.196247 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.196249 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.196252 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.196308 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.196351 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260927 03:18:07.197037 22573 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42697
I20260927 03:18:07.197124 22573 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:07.197228 22573 heartbeater.cc:511] Master 127.17.52.190:42697 requested a full tablet report, sending...
I20260927 03:18:07.197556 22251 ts_manager.cc:194] Registered new tserver with Master: 5fe578df15b649b98d8b740bac7c66e1 (127.17.52.130:36245)
I20260927 03:18:07.201033 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22444
I20260927 03:18:07.201130 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:07.201304 17618 external_mini_cluster.cc:1428] Setting key e71ad6176be831df3d8571a485f43897
I20260927 03:18:07.202145 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--builtin_ntp_servers=127.17.52.148:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:18:07.281901 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:07.282109 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:07.282131 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:07.283305 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:07.283347 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:07.283372 22576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:07.284825 22576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:07.285079 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:07.285375 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:07.290571 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.290606 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.290580 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.290969 22576 server_base.cc:1034] running on GCE node
I20260927 03:18:07.291177 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:07.291383 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:07.292507 22576 hybrid_clock.cc:648] HybridClock initialized: now 1790479087292491 us; error 28 us; skew 500 ppm
I20260927 03:18:07.293803 22576 webserver.cc:460] Webserver started at http://127.17.52.131:41489/ using document root <none> and password file <none>
I20260927 03:18:07.294041 22576 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:07.294088 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:07.294253 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:07.295399 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "e36ed11464c84a2797e623c89d40b8b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "118070ef2d3471b129d47386d3e0ad10"
server_key_iv: "9468158ca3cd755e611cb95e9c295f15"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.295876 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "e36ed11464c84a2797e623c89d40b8b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "118070ef2d3471b129d47386d3e0ad10"
server_key_iv: "9468158ca3cd755e611cb95e9c295f15"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.297341 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260927 03:18:07.298034 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.298209 22576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.298309 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "e36ed11464c84a2797e623c89d40b8b4"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "118070ef2d3471b129d47386d3e0ad10"
server_key_iv: "9468158ca3cd755e611cb95e9c295f15"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.298398 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:07.301813 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:07.302074 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:07.302217 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:07.302433 22576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:07.302736 22576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:07.302790 22576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.302829 22576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:07.302857 22576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.308516 22576 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:36271
I20260927 03:18:07.308557 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:36271 every 8 connection(s)
I20260927 03:18:07.308900 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:07.311908 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.309608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:58439 (local address 127.17.52.190:42697)
0927 03:18:07.309673 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.309675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.309900 (+ 225us) server_negotiation.cc:407] Connection header received
0927 03:18:07.310391 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.310392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.310399 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.310458 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.311648 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.311650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.311650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.311654 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.311669 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.311673 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.311721 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.311823 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.311838 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:18:07.312104 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.309617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:42697 (local address 127.17.52.131:58439)
0927 03:18:07.309776 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.309809 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.310284 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.310535 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.310538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.310546 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.311494 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.311495 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.311522 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.311549 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.311728 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.311733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.311736 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.311831 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.311891 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260927 03:18:07.312618 22704 heartbeater.cc:347] Connected to a master server at 127.17.52.190:42697
I20260927 03:18:07.312707 22704 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:07.312829 22704 heartbeater.cc:511] Master 127.17.52.190:42697 requested a full tablet report, sending...
I20260927 03:18:07.313125 22251 ts_manager.cc:194] Registered new tserver with Master: e36ed11464c84a2797e623c89d40b8b4 (127.17.52.131:36271)
I20260927 03:18:07.316834 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22576
I20260927 03:18:07.316910 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:07.317040 17618 external_mini_cluster.cc:1428] Setting key 3baa5ac5071e5b9b03fe59acf9ca873a
I20260927 03:18:07.317799 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:18:07.319932 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.319023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.17.52.190:42697)
0927 03:18:07.319086 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.319089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.319168 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:18:07.319213 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.319214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.319218 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.319269 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.319708 (+ 439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.319710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.319710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.319713 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.319725 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.319728 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.319784 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.319819 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.319830 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:18:07.321590 22251 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:07.323040 22251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:07.326668 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:36245 (local address 127.0.0.1:39438)
0927 03:18:07.325172 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.325210 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.325261 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.325573 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.325578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.325583 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.326046 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.326046 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.326061 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.326086 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.326487 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.326493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.326496 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.326584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.326595 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260927 03:18:07.326803 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.17.52.130:36245)
0927 03:18:07.325426 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.325430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.325438 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:07.325458 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.325460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.325467 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.325573 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.326324 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.326332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.326333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.326338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.326377 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.326400 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.326438 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.326610 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.326629 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260927 03:18:07.327301 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33573 (local address 127.0.0.1:53188)
0927 03:18:07.325322 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.325331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.325365 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.326254 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.326262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.326265 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.326686 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.326686 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.326693 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.326704 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.327159 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.327160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.327161 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.327218 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.327231 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260927 03:18:07.327319 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.17.52.129:33573)
0927 03:18:07.326130 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.326134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.326145 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:07.326171 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.326173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.326183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.326253 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.327001 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.327009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.327009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.327013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.327053 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.327071 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.327154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.327221 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.327236 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":94,"threads_started":1}
I20260927 03:18:07.327411 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:36271 (local address 127.0.0.1:47348)
0927 03:18:07.325941 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.325951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.326005 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.326406 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.326407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.326409 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.326867 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.326867 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.326872 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.326880 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.327286 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.327287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.327287 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.327330 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.327341 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":75,"threads_started":1}
I20260927 03:18:07.327591 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.325410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.17.52.131:36271)
0927 03:18:07.325568 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.325571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.326019 (+ 448us) server_negotiation.cc:407] Connection header received
0927 03:18:07.326053 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.326056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.326064 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.326142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.327117 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.327140 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.327140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.327145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.327178 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.327200 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.327254 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.327499 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.327516 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260927 03:18:07.327551 22506 tablet_service.cc:1467] Processing CreateTablet for tablet dec3a80419814799ae9501204e560681 (DEFAULT_TABLE table=test-table [id=1d28ef21d5884a31953ab3b4fb20a5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:07.327911 22506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3a80419814799ae9501204e560681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:07.327894 22374 tablet_service.cc:1467] Processing CreateTablet for tablet dec3a80419814799ae9501204e560681 (DEFAULT_TABLE table=test-table [id=1d28ef21d5884a31953ab3b4fb20a5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:07.328094 22638 tablet_service.cc:1467] Processing CreateTablet for tablet dec3a80419814799ae9501204e560681 (DEFAULT_TABLE table=test-table [id=1d28ef21d5884a31953ab3b4fb20a5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:07.328186 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3a80419814799ae9501204e560681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:07.328327 22638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3a80419814799ae9501204e560681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:07.330626 22718 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Bootstrap starting.
I20260927 03:18:07.331144 22719 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Bootstrap starting.
I20260927 03:18:07.331281 22718 tablet_bootstrap.cc:654] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:07.331463 22720 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: Bootstrap starting.
I20260927 03:18:07.331588 22718 log.cc:824] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:07.331772 22719 tablet_bootstrap.cc:654] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:07.332043 22719 log.cc:824] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:07.332119 22720 tablet_bootstrap.cc:654] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:07.332392 22720 log.cc:824] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:07.332784 22719 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: No bootstrap required, opened a new log
I20260927 03:18:07.332898 22719 ts_tablet_manager.cc:1397] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:18:07.333135 22720 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: No bootstrap required, opened a new log
I20260927 03:18:07.333185 22720 ts_tablet_manager.cc:1397] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:18:07.334941 22719 raft_consensus.cc:348] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.335089 22719 raft_consensus.cc:374] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:07.335134 22719 raft_consensus.cc:729] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe578df15b649b98d8b740bac7c66e1, State: Initialized, Role: FOLLOWER
I20260927 03:18:07.335163 22720 raft_consensus.cc:348] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.335310 22720 raft_consensus.cc:374] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:07.335255 22719 consensus_queue.cc:260] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.335335 22720 raft_consensus.cc:729] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d311ae7042e47b1b09dd6ebec4464b4, State: Initialized, Role: FOLLOWER
I20260927 03:18:07.335493 22720 consensus_queue.cc:260] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.335546 22719 ts_tablet_manager.cc:1428] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:18:07.335580 22573 heartbeater.cc:503] Master 127.17.52.190:42697 was elected leader, sending a full tablet report...
I20260927 03:18:07.335731 22720 ts_tablet_manager.cc:1428] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:18:07.335803 22441 heartbeater.cc:503] Master 127.17.52.190:42697 was elected leader, sending a full tablet report...
I20260927 03:18:07.336160 22718 tablet_bootstrap.cc:492] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: No bootstrap required, opened a new log
I20260927 03:18:07.336231 22718 ts_tablet_manager.cc:1397] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 03:18:07.338270 22718 raft_consensus.cc:348] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.338390 22718 raft_consensus.cc:374] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:07.338411 22718 raft_consensus.cc:729] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36ed11464c84a2797e623c89d40b8b4, State: Initialized, Role: FOLLOWER
I20260927 03:18:07.338482 22718 consensus_queue.cc:260] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.338655 22718 ts_tablet_manager.cc:1428] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:18:07.338711 22704 heartbeater.cc:503] Master 127.17.52.190:42697 was elected leader, sending a full tablet report...
I20260927 03:18:07.397883 22726 raft_consensus.cc:482] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:07.398039 22726 raft_consensus.cc:504] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.398463 22726 leader_election.cc:288] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d311ae7042e47b1b09dd6ebec4464b4 (127.17.52.129:33573), 5fe578df15b649b98d8b740bac7c66e1 (127.17.52.130:36245)
I20260927 03:18:07.399786 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.398659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33573 (local address 127.17.52.131:53777)
0927 03:18:07.398713 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.398724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.398765 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.398953 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.398955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.398958 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.399443 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.399443 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.399450 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.399460 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.399654 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.399655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.399655 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.399705 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.399720 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:18:07.399796 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.398713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:53777 (local address 127.17.52.129:33573)
0927 03:18:07.398758 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.398760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.398769 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:07.398843 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.398845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.398850 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.398910 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.399547 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.399548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.399548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.399553 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.399576 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.399580 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.399655 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.399707 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.399721 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:07.399943 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.398793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:43167 (local address 127.17.52.130:36245)
0927 03:18:07.398838 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.398839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.398961 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:18:07.398992 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.398993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.398999 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.399067 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.399728 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.399730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.399730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.399736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.399751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.399755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.399814 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.399866 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.399879 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:18:07.399953 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.398698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:36245 (local address 127.17.52.131:43167)
0927 03:18:07.398882 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.398892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.398941 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.399077 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.399077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.399080 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:07.399622 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:07.399622 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:07.399629 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:07.399640 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:07.399813 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:07.399814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:07.399814 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:18:07.399857 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:18:07.399870 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":134,"threads_started":1}
I20260927 03:18:07.400171 22394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3a80419814799ae9501204e560681" candidate_uuid: "e36ed11464c84a2797e623c89d40b8b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" is_pre_election: true
I20260927 03:18:07.400353 22394 raft_consensus.cc:2454] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36ed11464c84a2797e623c89d40b8b4 in term 0.
I20260927 03:18:07.400334 22526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3a80419814799ae9501204e560681" candidate_uuid: "e36ed11464c84a2797e623c89d40b8b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe578df15b649b98d8b740bac7c66e1" is_pre_election: true
I20260927 03:18:07.400503 22526 raft_consensus.cc:2454] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36ed11464c84a2797e623c89d40b8b4 in term 0.
I20260927 03:18:07.400627 22594 leader_election.cc:302] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d311ae7042e47b1b09dd6ebec4464b4, e36ed11464c84a2797e623c89d40b8b4; no voters:
I20260927 03:18:07.400815 22726 raft_consensus.cc:2781] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:07.400859 22726 raft_consensus.cc:482] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:07.400875 22726 raft_consensus.cc:3037] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:07.401603 22726 raft_consensus.cc:504] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.401741 22726 leader_election.cc:288] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [CANDIDATE]: Term 1 election: Requested vote from peers 5d311ae7042e47b1b09dd6ebec4464b4 (127.17.52.129:33573), 5fe578df15b649b98d8b740bac7c66e1 (127.17.52.130:36245)
I20260927 03:18:07.401895 22526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3a80419814799ae9501204e560681" candidate_uuid: "e36ed11464c84a2797e623c89d40b8b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe578df15b649b98d8b740bac7c66e1"
I20260927 03:18:07.401906 22394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3a80419814799ae9501204e560681" candidate_uuid: "e36ed11464c84a2797e623c89d40b8b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d311ae7042e47b1b09dd6ebec4464b4"
I20260927 03:18:07.401970 22526 raft_consensus.cc:3037] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:07.401989 22394 raft_consensus.cc:3037] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:07.402848 22526 raft_consensus.cc:2454] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36ed11464c84a2797e623c89d40b8b4 in term 1.
I20260927 03:18:07.402848 22394 raft_consensus.cc:2454] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36ed11464c84a2797e623c89d40b8b4 in term 1.
I20260927 03:18:07.403040 22594 leader_election.cc:302] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d311ae7042e47b1b09dd6ebec4464b4, e36ed11464c84a2797e623c89d40b8b4; no voters:
I20260927 03:18:07.403131 22726 raft_consensus.cc:2781] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:07.403299 22726 raft_consensus.cc:686] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 1 LEADER]: Becoming Leader. State: Replica: e36ed11464c84a2797e623c89d40b8b4, State: Running, Role: LEADER
I20260927 03:18:07.403407 22726 consensus_queue.cc:237] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } }
I20260927 03:18:07.404290 22251 catalog_manager.cc:5495] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e36ed11464c84a2797e623c89d40b8b4 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "e36ed11464c84a2797e623c89d40b8b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e36ed11464c84a2797e623c89d40b8b4" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 36271 } health_report { overall_health: HEALTHY } } }
I20260927 03:18:07.423707 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.422625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.17.52.131:36271)
0927 03:18:07.422669 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.422670 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.422680 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:07.422759 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.422760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.422780 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.422844 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.423477 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.423479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.423479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.423484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.423499 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.423504 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.423555 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.423619 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.423632 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:07.425596 22394 raft_consensus.cc:1261] T dec3a80419814799ae9501204e560681 P 5d311ae7042e47b1b09dd6ebec4464b4 [term 1 FOLLOWER]: Refusing update from remote peer e36ed11464c84a2797e623c89d40b8b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:07.425617 22526 raft_consensus.cc:1261] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 1 FOLLOWER]: Refusing update from remote peer e36ed11464c84a2797e623c89d40b8b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:07.425977 22726 consensus_queue.cc:1035] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d311ae7042e47b1b09dd6ebec4464b4" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:07.426137 22728 consensus_queue.cc:1035] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fe578df15b649b98d8b740bac7c66e1" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:07.428357 22731 mvcc.cc:205] Tried to move back new op lower bound from 7333802342092386304 to 7333802342005182464. Current Snapshot: MvccSnapshot[applied={T|T < 7333802342092386304}]
I20260927 03:18:07.429029 22733 mvcc.cc:205] Tried to move back new op lower bound from 7333802342092386304 to 7333802342005182464. Current Snapshot: MvccSnapshot[applied={T|T < 7333802342092386304}]
I20260927 03:18:07.429425 22732 mvcc.cc:205] Tried to move back new op lower bound from 7333802342092386304 to 7333802342005182464. Current Snapshot: MvccSnapshot[applied={T|T < 7333802342092386304 or (T in {7333802342092386304})}]
I20260927 03:18:07.430724 22251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:07.430820 22251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:07.431591 22251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05182c9a14a94acc961a6dd2b0020df3: Sending DeleteTablet for 3 replicas of tablet dec3a80419814799ae9501204e560681
I20260927 03:18:07.431937 22638 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3a80419814799ae9501204e560681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:07 UTC) from {username='slave'} at 127.0.0.1:47348
I20260927 03:18:07.431978 22374 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3a80419814799ae9501204e560681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:07 UTC) from {username='slave'} at 127.0.0.1:53188
I20260927 03:18:07.432236 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22312
I20260927 03:18:07.432345 22506 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3a80419814799ae9501204e560681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:07 UTC) from {username='slave'} at 127.0.0.1:39438
I20260927 03:18:07.432891 22743 tablet_replica.cc:330] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: stopping tablet replica
I20260927 03:18:07.432982 22743 raft_consensus.cc:2229] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:07.433082 22743 raft_consensus.cc:2258] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:07.433399 22741 tablet_replica.cc:330] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: stopping tablet replica
I20260927 03:18:07.433512 22741 raft_consensus.cc:2229] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:07.433507 22743 ts_tablet_manager.cc:1905] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:07.433600 22741 raft_consensus.cc:2258] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:07.435427 22743 ts_tablet_manager.cc:1918] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:07.435508 22743 log.cc:1196] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/dec3a80419814799ae9501204e560681
I20260927 03:18:07.435758 22743 ts_tablet_manager.cc:1939] T dec3a80419814799ae9501204e560681 P 5fe578df15b649b98d8b740bac7c66e1: Deleting consensus metadata
I20260927 03:18:07.436193 22235 catalog_manager.cc:4841] TS 5fe578df15b649b98d8b740bac7c66e1 (127.17.52.130:36245): tablet dec3a80419814799ae9501204e560681 (table test-table [id=1d28ef21d5884a31953ab3b4fb20a5b9]) successfully deleted
I20260927 03:18:07.437103 22741 ts_tablet_manager.cc:1905] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:18:07.438573 22238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:33573 (error 108)
I20260927 03:18:07.438735 22741 ts_tablet_manager.cc:1918] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:07.438818 22741 log.cc:1196] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/dec3a80419814799ae9501204e560681
I20260927 03:18:07.438838 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22444
I20260927 03:18:07.439091 22741 ts_tablet_manager.cc:1939] T dec3a80419814799ae9501204e560681 P e36ed11464c84a2797e623c89d40b8b4: Deleting consensus metadata
I20260927 03:18:07.440217 22238 catalog_manager.cc:4841] TS e36ed11464c84a2797e623c89d40b8b4 (127.17.52.131:36271): tablet dec3a80419814799ae9501204e560681 (table test-table [id=1d28ef21d5884a31953ab3b4fb20a5b9]) successfully deleted
I20260927 03:18:07.444198 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.444046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33573 (local address 127.0.0.1:53196)
0927 03:18:07.444114 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.444161 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 03:18:07.444233 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22576
W20260927 03:18:07.444329 22238 catalog_manager.cc:4584] TS 5d311ae7042e47b1b09dd6ebec4464b4 (127.17.52.129:33573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dec3a80419814799ae9501204e560681: Network error: Client connection negotiation failed: client connection to 127.17.52.129:33573: connect: Connection refused (error 111)
I20260927 03:18:07.449149 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22220
2026-09-27T03:18:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (634 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 03:18:07.468737 17618 mini_oidc.cc:152] Starting JWKS server
I20260927 03:18:07.469952 17618 webserver.cc:460] Webserver started at https://127.0.0.1:37887/ using document root <none> and password file <none>
I20260927 03:18:07.470031 17618 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:18:07.470156 17618 webserver.cc:460] Webserver started at http://127.0.0.1:34165/ using document root <none> and password file <none>
2026-09-27T03:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:07Z Disabled control of system clock
I20260927 03:18:07.478008 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:37767
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:37767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37887/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/testchainca.cert with env {}
W20260927 03:18:07.558821 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:07.559033 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:07.559056 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:07.560177 22750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:07.560220 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:07.560232 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:07.560245 22750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:07.560257 22750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:18:07.560269 22750 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:18:07.560280 22750 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37887/jwks.json
I20260927 03:18:07.561934 22750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:37767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37887/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:07.562140 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:07.562311 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:07.566947 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.566977 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.566963 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.567289 22750 server_base.cc:1034] running on GCE node
I20260927 03:18:07.567544 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:07.567850 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:07.569022 22750 hybrid_clock.cc:648] HybridClock initialized: now 1790479087569008 us; error 29 us; skew 500 ppm
I20260927 03:18:07.570223 22750 webserver.cc:460] Webserver started at http://127.17.52.190:45611/ using document root <none> and password file <none>
I20260927 03:18:07.570461 22750 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:07.570529 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:07.570676 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:07.571707 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "c8f6397aba274232801d18d6d5f69759"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "33871d92b4fa92889398ad67b8da387b"
server_key_iv: "d221e597e532cf81ea88175f58fedf6d"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.572060 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "c8f6397aba274232801d18d6d5f69759"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "33871d92b4fa92889398ad67b8da387b"
server_key_iv: "d221e597e532cf81ea88175f58fedf6d"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.573400 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:07.574100 22764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.574275 22750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:07.574378 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "c8f6397aba274232801d18d6d5f69759"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "33871d92b4fa92889398ad67b8da387b"
server_key_iv: "d221e597e532cf81ea88175f58fedf6d"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.574450 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:07.596853 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:07.597173 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:07.597332 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:07.601181 22750 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:37767
I20260927 03:18:07.601230 22816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:37767 every 8 connection(s)
I20260927 03:18:07.601598 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:07.602111 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22750
I20260927 03:18:07.602222 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:07.602411 17618 external_mini_cluster.cc:1428] Setting key 19ad37b89ed0b8a2b9b2874d92f01251
I20260927 03:18:07.602720 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:07.606091 22817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759: Bootstrap starting.
I20260927 03:18:07.606776 22817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:07.607080 22817 log.cc:824] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:07.607815 22817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759: No bootstrap required, opened a new log
I20260927 03:18:07.608623 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.602969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.17.52.190:37767)
0927 03:18:07.603195 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.603204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.603236 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:18:07.603862 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.603871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.603954 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.604105 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.604267 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.605640 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.606450 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.607085 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.607218 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.607948 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.607952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.607952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.608132 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.608163 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.608173 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.608249 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.608319 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.608395 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260927 03:18:07.609618 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:18:07.610059 22817 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } }
I20260927 03:18:07.610209 22817 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:07.610270 22817 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f6397aba274232801d18d6d5f69759, State: Initialized, Role: FOLLOWER
I20260927 03:18:07.610392 22817 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } }
I20260927 03:18:07.610474 22817 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:07.610496 22817 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:07.610564 22817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:07.611073 22817 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } }
I20260927 03:18:07.611156 22817 leader_election.cc:302] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f6397aba274232801d18d6d5f69759; no voters:
I20260927 03:18:07.611338 22817 leader_election.cc:288] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:07.611443 22822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:07.611581 22817 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:07.611605 22822 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [term 1 LEADER]: Becoming Leader. State: Replica: c8f6397aba274232801d18d6d5f69759, State: Running, Role: LEADER
I20260927 03:18:07.611680 22822 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } }
I20260927 03:18:07.612062 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f6397aba274232801d18d6d5f69759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } } }
I20260927 03:18:07.612143 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:07.612313 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f6397aba274232801d18d6d5f69759. Latest consensus state: current_term: 1 leader_uuid: "c8f6397aba274232801d18d6d5f69759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f6397aba274232801d18d6d5f69759" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37767 } } }
I20260927 03:18:07.612392 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:07.612793 22830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:07.613270 22830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:07.614713 22830 catalog_manager.cc:1347] Generated new cluster ID: 12527a80c06340cfb93a90041215abc8
I20260927 03:18:07.614773 22830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:07.638427 22830 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:07.639016 22830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:07.648459 22830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f6397aba274232801d18d6d5f69759: Generated new TSK 0
I20260927 03:18:07.648680 22830 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:18:07.675534 22847 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 03:18:07.675652 22847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:18:07.675870 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.656769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.17.52.190:37767)
0927 03:18:07.656826 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.656828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.656986 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:18:07.657089 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.657090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.657103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.657168 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:18:07.657363 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.657846 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.658640 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.658842 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.658924 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.659148 (+ 224us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:18:07.675628 (+ 16480us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:18:07.675798 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 03:18:07.675938 22820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 03:18:07.676309 22846 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:07.656685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:37767 (local address 127.0.0.1:52488)
0927 03:18:07.656915 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.656925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.657007 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.657174 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.657175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.657177 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:18:07.657250 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:07.657252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.657948 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.657949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:07.658554 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:07.658556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.658927 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.658928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:07.658995 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.659022 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:18:07.675843 (+ 16821us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:18:07.675991 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.52.190:37767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.52.190:37767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 03:18:07.676882 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22750
2026-09-27T03:18:07Z chronyd exiting
I20260927 03:18:07.712965 17618 test_util.cc:175] -----------------------------------------------
I20260927 03:18:07.713070 17618 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479066103559-17618-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 03:18:07.713827 17618 mini_oidc.cc:152] Starting JWKS server
I20260927 03:18:07.714524 17618 webserver.cc:460] Webserver started at https://127.0.0.1:46629/ using document root <none> and password file <none>
I20260927 03:18:07.714591 17618 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:18:07.714782 17618 webserver.cc:460] Webserver started at http://127.0.0.1:41393/ using document root <none> and password file <none>
2026-09-27T03:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:07Z Disabled control of system clock
I20260927 03:18:07.722329 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40379
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:40379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/testchainca.cert with env {}
W20260927 03:18:07.801236 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:07.801442 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:07.801465 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:07.802644 22853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:07.802686 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:07.802699 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:07.802713 22853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:07.802726 22853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:18:07.802786 22853 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:18:07.802804 22853 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46629/jwks.json
I20260927 03:18:07.804325 22853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:40379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:07.804613 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:07.804847 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:07.809806 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:07.809754 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.809906 22853 server_base.cc:1034] running on GCE node
W20260927 03:18:07.809754 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:07.810379 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:07.810745 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:07.811896 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790479087811872 us; error 35 us; skew 500 ppm
I20260927 03:18:07.813122 22853 webserver.cc:460] Webserver started at http://127.17.52.190:42975/ using document root <none> and password file <none>
I20260927 03:18:07.813364 22853 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:07.813416 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:07.813580 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:07.814637 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "20079e958e4241799369364df09683c3"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "859b89e04bf259b13bb57e4db7deb690"
server_key_iv: "e13c3542d437e7fe316960846104a781"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.815009 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "20079e958e4241799369364df09683c3"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "859b89e04bf259b13bb57e4db7deb690"
server_key_iv: "e13c3542d437e7fe316960846104a781"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.816352 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:07.817173 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.817354 22853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:07.817442 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "20079e958e4241799369364df09683c3"
format_stamp: "Formatted at 2026-09-27 03:18:07 on dist-test-slave-7j43"
server_key: "859b89e04bf259b13bb57e4db7deb690"
server_key_iv: "e13c3542d437e7fe316960846104a781"
server_key_version: "encryptionkey@0"
I20260927 03:18:07.817498 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:07.837328 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:07.837617 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:07.837721 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:07.841581 22853 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:40379
I20260927 03:18:07.841637 22919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:40379 every 8 connection(s)
I20260927 03:18:07.842001 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:07.842710 22920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:07.845527 22920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: Bootstrap starting.
I20260927 03:18:07.846120 22920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:07.846427 22920 log.cc:824] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:07.846479 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22853
I20260927 03:18:07.846647 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:07.846866 17618 external_mini_cluster.cc:1428] Setting key afb1a3ca61d8739b119f54679df49cba
I20260927 03:18:07.847187 22920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: No bootstrap required, opened a new log
I20260927 03:18:07.849035 22920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } }
I20260927 03:18:07.849198 22920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:07.849241 22920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20079e958e4241799369364df09683c3, State: Initialized, Role: FOLLOWER
I20260927 03:18:07.849393 22920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } }
I20260927 03:18:07.849494 22920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:07.849540 22920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:07.849593 22920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:07.850155 22920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } }
I20260927 03:18:07.850276 22920 leader_election.cc:302] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20079e958e4241799369364df09683c3; no voters:
I20260927 03:18:07.850471 22920 leader_election.cc:288] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:07.850493 22925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:07.850621 22925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [term 1 LEADER]: Becoming Leader. State: Replica: 20079e958e4241799369364df09683c3, State: Running, Role: LEADER
I20260927 03:18:07.850708 22925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } }
I20260927 03:18:07.850771 22920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:07.851027 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20079e958e4241799369364df09683c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } } }
I20260927 03:18:07.851168 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:07.851032 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20079e958e4241799369364df09683c3. Latest consensus state: current_term: 1 leader_uuid: "20079e958e4241799369364df09683c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20079e958e4241799369364df09683c3" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40379 } } }
I20260927 03:18:07.851468 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3 [sys.catalog]: This master's current role is: LEADER
W20260927 03:18:07.852746 22940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:18:07.852865 22940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:18:07.852965 22932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:07.853134 22932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:07.854385 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.847417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.17.52.190:40379)
0927 03:18:07.847675 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.847684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.847719 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:18:07.849020 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.849029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.849097 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.849268 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:07.849488 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.851075 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.852049 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.852690 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.853133 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.853572 (+ 439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:07.853576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:07.853576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:07.853791 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:07.853835 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:07.853850 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:07.853945 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:07.854024 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:18:07.854108 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260927 03:18:07.854861 22932 catalog_manager.cc:1347] Generated new cluster ID: bd7c8975b0174b10bf6fdb461531cc11
I20260927 03:18:07.854964 22932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:07.855367 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:18:07.865720 22932 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:07.866238 22932 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:07.870595 22932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20079e958e4241799369364df09683c3: Generated new TSK 0
I20260927 03:18:07.870779 22932 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:18:07.898090 22950 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 03:18:07.898181 22950 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:18:07.898515 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:07.879184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.17.52.190:40379)
0927 03:18:07.879227 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:07.879229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:07.879332 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:18:07.879405 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:07.879406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:07.879416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:07.879478 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:18:07.879631 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.880135 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.881287 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.881507 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.881572 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.881781 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:18:07.898332 (+ 16551us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:18:07.898463 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:18:07.898589 22924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 03:18:07.898741 22949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:07.879069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40379 (local address 127.0.0.1:34506)
0927 03:18:07.879245 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:07.879264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:07.879334 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:07.879498 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:07.879499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:07.879501 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:18:07.879562 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:07.879564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.880217 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.880218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:07.881180 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:07.881183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:07.881602 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:07.881603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:07.881668 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:07.881674 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:18:07.898471 (+ 16797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:18:07.898623 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.52.190:40379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260927 03:18:07.899118 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22853
2026-09-27T03:18:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 03:18:07.936806 17618 mini_oidc.cc:152] Starting JWKS server
I20260927 03:18:07.937505 17618 webserver.cc:460] Webserver started at https://127.0.0.1:45381/ using document root <none> and password file <none>
I20260927 03:18:07.937551 17618 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:18:07.937721 17618 webserver.cc:460] Webserver started at http://127.0.0.1:37495/ using document root <none> and password file <none>
2026-09-27T03:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:07Z Disabled control of system clock
I20260927 03:18:07.945374 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43743
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:43743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:18:08.025888 22956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:08.026093 22956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:08.026117 22956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:08.027226 22956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:08.027261 22956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:08.027272 22956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:08.027285 22956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:08.027297 22956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:18:08.027308 22956 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:18:08.027320 22956 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45381/jwks.json
I20260927 03:18:08.028674 22956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:43743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:08.028863 22956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:08.029162 22956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:08.034132 22964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:08.034145 22962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:08.034168 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:08.034461 22956 server_base.cc:1034] running on GCE node
I20260927 03:18:08.034706 22956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:08.034996 22956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:08.036170 22956 hybrid_clock.cc:648] HybridClock initialized: now 1790479088036156 us; error 43 us; skew 500 ppm
I20260927 03:18:08.037390 22956 webserver.cc:460] Webserver started at http://127.17.52.190:42539/ using document root <none> and password file <none>
I20260927 03:18:08.037637 22956 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:08.037721 22956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:08.037866 22956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:08.038936 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "8ca68051cef74e9db856d2360f8f23eb"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "dea963caf7d33dcf39bd3b4a80a097b0"
server_key_iv: "15e2e2d238b1b8eb3e44ebfb82ef8c47"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.039311 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "8ca68051cef74e9db856d2360f8f23eb"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "dea963caf7d33dcf39bd3b4a80a097b0"
server_key_iv: "15e2e2d238b1b8eb3e44ebfb82ef8c47"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.040545 22956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:08.041316 22970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:08.041479 22956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:08.041574 22956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "8ca68051cef74e9db856d2360f8f23eb"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "dea963caf7d33dcf39bd3b4a80a097b0"
server_key_iv: "15e2e2d238b1b8eb3e44ebfb82ef8c47"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.041656 22956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:08.059183 22956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:08.059487 22956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:08.059635 22956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:08.063628 22956 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:43743
I20260927 03:18:08.063661 23022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:43743 every 8 connection(s)
I20260927 03:18:08.064002 22956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:08.064776 23023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:08.067440 23023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb: Bootstrap starting.
I20260927 03:18:08.068006 23023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:08.068288 23023 log.cc:824] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:08.069011 23023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb: No bootstrap required, opened a new log
I20260927 03:18:08.069605 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 22956
I20260927 03:18:08.069707 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:08.069881 17618 external_mini_cluster.cc:1428] Setting key f48349e0ddf917e513971160aa8abd9a
I20260927 03:18:08.071133 23023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } }
I20260927 03:18:08.071262 23023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:08.071309 23023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca68051cef74e9db856d2360f8f23eb, State: Initialized, Role: FOLLOWER
I20260927 03:18:08.071445 23023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } }
I20260927 03:18:08.071529 23023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:08.071579 23023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:08.071625 23023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:08.072166 23023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } }
I20260927 03:18:08.072278 23023 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca68051cef74e9db856d2360f8f23eb; no voters:
I20260927 03:18:08.072463 23023 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:08.072484 23028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:08.072616 23028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [term 1 LEADER]: Becoming Leader. State: Replica: 8ca68051cef74e9db856d2360f8f23eb, State: Running, Role: LEADER
I20260927 03:18:08.072700 23028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } }
I20260927 03:18:08.072746 23023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:08.072985 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca68051cef74e9db856d2360f8f23eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } } }
I20260927 03:18:08.073065 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:08.073218 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca68051cef74e9db856d2360f8f23eb. Latest consensus state: current_term: 1 leader_uuid: "8ca68051cef74e9db856d2360f8f23eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca68051cef74e9db856d2360f8f23eb" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43743 } } }
I20260927 03:18:08.073292 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:08.073264 23032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:08.073992 23032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:08.076035 23032 catalog_manager.cc:1347] Generated new cluster ID: 7046d3d27be442898afc9dff4ba0ca48
I20260927 03:18:08.076088 23032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:08.076563 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:08.070349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.17.52.190:43743)
0927 03:18:08.070538 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:08.070545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:08.070568 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:18:08.071113 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:08.071119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:08.071197 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:08.071339 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:08.071580 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.073214 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.074515 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.074895 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.074975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.075739 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:08.075743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:08.075743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:08.075938 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:08.075980 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:08.075994 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:08.076089 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:08.076343 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 03:18:08.076407 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260927 03:18:08.077567 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:18:08.094425 23032 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:08.094957 23032 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:08.102823 23032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca68051cef74e9db856d2360f8f23eb: Generated new TSK 0
I20260927 03:18:08.103015 23032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:08.119622 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:08.115981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.17.52.190:43743)
0927 03:18:08.116030 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:08.116032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:08.116094 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:18:08.116190 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:08.116190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:08.116201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:08.116269 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:18:08.116387 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.116933 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.118080 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.118266 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.118318 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.118496 (+ 178us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:18:08.119427 (+ 931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:18:08.119553 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 03:18:08.119688 23027 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 03:18:08.119741 23053 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:08.115873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43743 (local address 127.0.0.1:34618)
0927 03:18:08.116036 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:08.116047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:08.116112 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:08.116272 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:08.116273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:08.116275 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:18:08.116331 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:08.116333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.117006 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.117008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:08.117994 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:08.117996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.118339 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.118340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:08.118398 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.118403 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:18:08.119530 (+ 1127us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:18:08.119641 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.52.190:43743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260927 03:18:08.119958 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 22956
E20260927 03:18:08.130623 23054 webserver.cc:533] Webserver: sslize failed:
E20260927 03:18:08.130692 23054 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:18:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T03:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:08Z Disabled control of system clock
I20260927 03:18:08.164175 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38967
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38967
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/testchainca.cert with env {}
W20260927 03:18:08.244961 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:08.245159 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:08.245180 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:08.246244 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:08.246281 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:08.246294 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:08.246305 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:08.246317 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:18:08.246330 23058 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:18:08.246341 23058 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 03:18:08.247716 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:08.247965 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:08.248138 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:08.252838 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:08.252933 23058 server_base.cc:1034] running on GCE node
W20260927 03:18:08.252803 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:08.252821 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:08.253259 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:08.253544 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:08.254714 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790479088254692 us; error 38 us; skew 500 ppm
I20260927 03:18:08.255884 23058 webserver.cc:460] Webserver started at http://127.17.52.190:43955/ using document root <none> and password file <none>
I20260927 03:18:08.256120 23058 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:08.256209 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:08.256356 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:08.257498 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "d2f8094f748442c0be40bf223471fbf7"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "f2e9b2580babcc84e4eeef46af0a06df"
server_key_iv: "070aa73154dcd59e5382ce557a076a5c"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.257869 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "d2f8094f748442c0be40bf223471fbf7"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "f2e9b2580babcc84e4eeef46af0a06df"
server_key_iv: "070aa73154dcd59e5382ce557a076a5c"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.259150 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:08.259877 23072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:08.260051 23058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:08.260147 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "d2f8094f748442c0be40bf223471fbf7"
format_stamp: "Formatted at 2026-09-27 03:18:08 on dist-test-slave-7j43"
server_key: "f2e9b2580babcc84e4eeef46af0a06df"
server_key_iv: "070aa73154dcd59e5382ce557a076a5c"
server_key_version: "encryptionkey@0"
I20260927 03:18:08.260246 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:08.284119 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:08.284471 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:08.284696 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:08.288484 23058 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38967
I20260927 03:18:08.288527 23124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38967 every 8 connection(s)
I20260927 03:18:08.288929 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:08.289932 23125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:08.292131 23125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7: Bootstrap starting.
I20260927 03:18:08.292687 23125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:08.292939 23125 log.cc:824] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:08.293561 23125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7: No bootstrap required, opened a new log
I20260927 03:18:08.295089 23125 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } }
I20260927 03:18:08.295221 23125 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:08.295243 23125 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f8094f748442c0be40bf223471fbf7, State: Initialized, Role: FOLLOWER
I20260927 03:18:08.295393 23125 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } }
I20260927 03:18:08.295481 23125 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:08.295504 23125 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:08.295533 23125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:08.296082 23125 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } }
I20260927 03:18:08.296165 23125 leader_election.cc:302] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f8094f748442c0be40bf223471fbf7; no voters:
I20260927 03:18:08.296289 23125 leader_election.cc:288] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:08.296375 23128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:08.296525 23128 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [term 1 LEADER]: Becoming Leader. State: Replica: d2f8094f748442c0be40bf223471fbf7, State: Running, Role: LEADER
I20260927 03:18:08.296563 23125 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:08.296651 23128 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } }
I20260927 03:18:08.297029 23130 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f8094f748442c0be40bf223471fbf7. Latest consensus state: current_term: 1 leader_uuid: "d2f8094f748442c0be40bf223471fbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } } }
I20260927 03:18:08.297123 23130 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:08.297111 23129 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f8094f748442c0be40bf223471fbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f8094f748442c0be40bf223471fbf7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38967 } } }
I20260927 03:18:08.297219 23129 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:08.297358 23136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:08.297887 23136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:08.298682 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23058
I20260927 03:18:08.298799 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:08.299007 17618 external_mini_cluster.cc:1428] Setting key d8c398722181e6aecec4c56c85202cf5
I20260927 03:18:08.299512 23136 catalog_manager.cc:1347] Generated new cluster ID: 7cd141d07d624cc3a52ee2fb7a78a540
I20260927 03:18:08.299568 23136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:08.310182 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:08.299473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.17.52.190:38967)
0927 03:18:08.299706 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:08.299714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:08.304834 (+ 5120us) server_negotiation.cc:407] Connection header received
0927 03:18:08.304962 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:08.304969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:08.305038 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:08.305184 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:08.305410 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.306856 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.307973 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.308604 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.308709 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.309442 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:08.309445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:08.309446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:08.309686 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:08.309727 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:08.309740 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:08.309802 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:08.309885 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:18:08.309959 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260927 03:18:08.311110 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:18:08.317287 23136 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:08.317791 23136 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:08.321058 23136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f8094f748442c0be40bf223471fbf7: Generated new TSK 0
I20260927 03:18:08.321239 23136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:08.749468 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:08.325579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.17.52.190:38967)
0927 03:18:08.325642 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:08.325644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:08.325800 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:18:08.325831 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:08.325832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:08.325843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:08.325902 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:18:08.326047 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.326514 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.327626 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.327828 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.327882 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.328075 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:18:08.749234 (+421159us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:18:08.749408 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 03:18:08.749553 23148 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 03:18:08.749620 23154 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:08.325519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38967 (local address 127.0.0.1:50378)
0927 03:18:08.325693 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:08.325704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:08.325777 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:08.325917 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:08.325918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:08.325920 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:18:08.325985 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:08.325987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.326582 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.326583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:08.327537 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:08.327540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:08.327910 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:08.327912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:08.327976 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:08.327982 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:18:08.749396 (+421414us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:18:08.749518 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.52.190:38967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260927 03:18:08.749994 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23058
2026-09-27T03:18:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (610 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:08 dist-test-slave-7j43 krb5kdc[23166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:08 dist-test-slave-7j43 krb5kdc[23166](info): set up 2 sockets
Sep 27 03:18:08 dist-test-slave-7j43 krb5kdc[23166](info): commencing operation
krb5kdc: starting...
W20260927 03:18:09.957273 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:10.070922 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33201
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:10.149657 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:10.149883 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:10.149919 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:10.151079 23179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:10.151144 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:10.151167 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:10.151192 23179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:10.151223 23179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:10.152674 23179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:10.152941 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:10.153180 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:10.157810 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.157829 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.157859 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:10.157977 23179 server_base.cc:1034] running on GCE node
I20260927 03:18:10.158463 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:10.158783 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:10.159945 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790479090159941 us; error 38 us; skew 500 ppm
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.164837 23179 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:10.165297 23179 webserver.cc:460] Webserver started at http://127.17.52.190:35257/ using document root <none> and password file <none>
I20260927 03:18:10.165524 23179 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:10.165591 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:10.165742 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:10.166740 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "e61c6da77f894e59a2c3de0096d8a67c"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "f4706f2bc861ee08def2faccbc637a9e"
server_key_iv: "68589306578f9f4a518580627bcb06aa"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.167109 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "e61c6da77f894e59a2c3de0096d8a67c"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "f4706f2bc861ee08def2faccbc637a9e"
server_key_iv: "68589306578f9f4a518580627bcb06aa"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.168440 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:10.169281 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.169484 23179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:10.169580 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "e61c6da77f894e59a2c3de0096d8a67c"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "f4706f2bc861ee08def2faccbc637a9e"
server_key_iv: "68589306578f9f4a518580627bcb06aa"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.169664 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:10.186656 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:10.189060 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:10.189244 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:10.193145 23179 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33201
I20260927 03:18:10.193292 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33201 every 8 connection(s)
I20260927 03:18:10.193575 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:10.194461 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:10.195462 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23179
I20260927 03:18:10.195588 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:10.195772 17618 external_mini_cluster.cc:1428] Setting key de5a4501e24bc422f4d8d0e6964950b4
I20260927 03:18:10.197392 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c: Bootstrap starting.
I20260927 03:18:10.198045 23247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:10.198366 23247 log.cc:824] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:10.199080 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c: No bootstrap required, opened a new log
I20260927 03:18:10.201261 23247 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } }
I20260927 03:18:10.201416 23247 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:10.201440 23247 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61c6da77f894e59a2c3de0096d8a67c, State: Initialized, Role: FOLLOWER
I20260927 03:18:10.201574 23247 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } }
I20260927 03:18:10.201649 23247 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:10.201671 23247 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:10.201740 23247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:10.202322 23247 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } }
I20260927 03:18:10.202437 23247 leader_election.cc:302] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61c6da77f894e59a2c3de0096d8a67c; no voters:
I20260927 03:18:10.202603 23247 leader_election.cc:288] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:10.202675 23252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:10.202854 23252 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [term 1 LEADER]: Becoming Leader. State: Replica: e61c6da77f894e59a2c3de0096d8a67c, State: Running, Role: LEADER
I20260927 03:18:10.202881 23247 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:10.203004 23252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } }
I20260927 03:18:10.203382 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61c6da77f894e59a2c3de0096d8a67c. Latest consensus state: current_term: 1 leader_uuid: "e61c6da77f894e59a2c3de0096d8a67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } } }
I20260927 03:18:10.203452 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:10.203610 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61c6da77f894e59a2c3de0096d8a67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6da77f894e59a2c3de0096d8a67c" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33201 } } }
I20260927 03:18:10.203668 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:10.204011 23258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:10.204586 23258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:10.206451 23258 catalog_manager.cc:1347] Generated new cluster ID: 1b4b537496fa4bac871aaffd0aacd328
I20260927 03:18:10.206513 23258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:10.207417 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.196332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.17.52.190:33201)
0927 03:18:10.196530 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.196539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.196600 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:18:10.197129 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.197139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.197202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.197357 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.197952 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.199325 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.200049 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.200755 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.200887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.203596 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.203599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.203600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.203611 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.205691 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.206110 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.206120 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.206120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.206218 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.206459 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.206460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.206460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.206787 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.206900 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.207093 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.207221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260927 03:18:10.247370 23258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:10.247926 23258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:10.251579 23258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61c6da77f894e59a2c3de0096d8a67c: Generated new TSK 0
I20260927 03:18:10.251783 23258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:10.279599 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:10.373761 23273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:10.373965 23273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:10.373986 23273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:10.375066 23273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:10.375098 23273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:10.375123 23273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:10.376467 23273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:10.376701 23273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:10.376891 23273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:10.381664 23279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:10.381691 23273 server_base.cc:1034] running on GCE node
W20260927 03:18:10.381675 23281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.381661 23278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:10.382145 23273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:10.382390 23273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:10.383533 23273 hybrid_clock.cc:648] HybridClock initialized: now 1790479090383519 us; error 31 us; skew 500 ppm
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.386058 23273 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:10.386525 23273 webserver.cc:460] Webserver started at http://127.17.52.129:43811/ using document root <none> and password file <none>
I20260927 03:18:10.386750 23273 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:10.386817 23273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:10.386962 23273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:10.387971 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "57caf0a3d326411aa86fdc732cb4d0f0"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "64924f677e3e48fe213ca95f23c0c4fd"
server_key_iv: "fd328a6d2dd0171e1e3c874e24084e46"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.388329 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "57caf0a3d326411aa86fdc732cb4d0f0"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "64924f677e3e48fe213ca95f23c0c4fd"
server_key_iv: "fd328a6d2dd0171e1e3c874e24084e46"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.389708 23273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:18:10.390512 23288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.390679 23273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:10.390782 23273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "57caf0a3d326411aa86fdc732cb4d0f0"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "64924f677e3e48fe213ca95f23c0c4fd"
server_key_iv: "fd328a6d2dd0171e1e3c874e24084e46"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.390869 23273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:10.408057 23273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:10.409860 23273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:10.410040 23273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:10.410269 23273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:10.410566 23273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:10.410619 23273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.410658 23273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:10.410686 23273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.416432 23273 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:41731
I20260927 03:18:10.416479 23401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:41731 every 8 connection(s)
I20260927 03:18:10.416832 23273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:10.425110 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23273
I20260927 03:18:10.425208 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:10.425410 17618 external_mini_cluster.cc:1428] Setting key 4eb8654d541462d40b16837509eaeed7
I20260927 03:18:10.426026 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.417531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:52987 (local address 127.17.52.190:33201)
0927 03:18:10.417609 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.417611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.417750 (+ 139us) server_negotiation.cc:407] Connection header received
0927 03:18:10.418622 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.418624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.418634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.418683 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.419233 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.419750 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.420810 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.421041 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.421110 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.422969 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.422971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.422971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.422978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.424533 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.424965 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.424966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.424966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.425021 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.425341 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.425342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.425342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.425455 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.425543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.425889 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.425962 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:18:10.426319 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.417483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33201 (local address 127.17.52.129:52987)
0927 03:18:10.417635 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:10.417669 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:10.418535 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:10.418733 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:10.418736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:10.418974 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:10.419148 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.419152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.419827 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.419828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.420722 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.420724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.421120 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.421120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.421215 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.421631 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:10.421638 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:10.422884 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:10.424676 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.424681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.424682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.424876 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.425087 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.425088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.425088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.425228 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.425563 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:10.425570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:10.425718 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:18:10.425879 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 03:18:10.426049 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260927 03:18:10.426786 23402 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33201
I20260927 03:18:10.426893 23402 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:10.427078 23402 heartbeater.cc:511] Master 127.17.52.190:33201 requested a full tablet report, sending...
I20260927 03:18:10.427585 23211 ts_manager.cc:194] Registered new tserver with Master: 57caf0a3d326411aa86fdc732cb4d0f0 (127.17.52.129:41731)
I20260927 03:18:10.428184 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:52987
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:10.448956 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:10.528002 23407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:10.528188 23407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:10.528210 23407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:10.529325 23407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:10.529361 23407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:10.529386 23407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:10.530717 23407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:10.530885 23407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:10.531040 23407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:10.535636 23415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.535640 23412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.535634 23413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:10.536057 23407 server_base.cc:1034] running on GCE node
I20260927 03:18:10.536248 23407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:10.536428 23407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:10.537566 23407 hybrid_clock.cc:648] HybridClock initialized: now 1790479090537549 us; error 33 us; skew 500 ppm
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.539888 23407 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:10.540290 23407 webserver.cc:460] Webserver started at http://127.17.52.130:41389/ using document root <none> and password file <none>
I20260927 03:18:10.540499 23407 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:10.540565 23407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:10.540729 23407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:10.541693 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "2e9de65965fd4a78ac467d4fc5caa729"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "07364c32b745d77b995a6b6cd4294f4c"
server_key_iv: "980db1d92b7023ac4773f51fa4aae889"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.542049 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "2e9de65965fd4a78ac467d4fc5caa729"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "07364c32b745d77b995a6b6cd4294f4c"
server_key_iv: "980db1d92b7023ac4773f51fa4aae889"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.543429 23407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:10.544129 23422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.544293 23407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:10.544389 23407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "2e9de65965fd4a78ac467d4fc5caa729"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "07364c32b745d77b995a6b6cd4294f4c"
server_key_iv: "980db1d92b7023ac4773f51fa4aae889"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.544462 23407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:10.560969 23407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:10.562656 23407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:10.562811 23407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:10.563073 23407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:10.563385 23407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:10.563436 23407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.563476 23407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:10.563505 23407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.569196 23407 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:42201
I20260927 03:18:10.569238 23535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:42201 every 8 connection(s)
I20260927 03:18:10.569543 23407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:10.573199 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23407
I20260927 03:18:10.573333 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:10.573544 17618 external_mini_cluster.cc:1428] Setting key 2d1c66189d6ffd51b3704146fe036566
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:10.579108 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.570254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41673 (local address 127.17.52.190:33201)
0927 03:18:10.570323 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.570325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.570582 (+ 257us) server_negotiation.cc:407] Connection header received
0927 03:18:10.571471 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.571473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.571482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.571533 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.572182 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.572736 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.573893 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.574134 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.574201 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.576303 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.576305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.576305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.576312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.577759 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.578156 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.578157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.578158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.578209 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.578473 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.578474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.578474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.578589 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.578663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.578940 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.579019 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41}
I20260927 03:18:10.579286 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.570237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33201 (local address 127.17.52.130:41673)
0927 03:18:10.570414 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:10.570474 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:10.571385 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:10.571587 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:10.571590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:10.571914 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:10.572092 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.572096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.572820 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.572821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.573819 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.573821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.574216 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.574217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.574358 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.574905 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:10.574912 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:10.576213 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:10.577852 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.577858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.577861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.578069 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.578297 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.578298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.578298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.578406 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.578655 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:10.578660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:10.578799 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:18:10.578896 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:18:10.579046 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260927 03:18:10.579632 23536 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33201
I20260927 03:18:10.579720 23536 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:10.579913 23536 heartbeater.cc:511] Master 127.17.52.190:33201 requested a full tablet report, sending...
I20260927 03:18:10.580283 23211 ts_manager.cc:194] Registered new tserver with Master: 2e9de65965fd4a78ac467d4fc5caa729 (127.17.52.130:42201)
I20260927 03:18:10.580734 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:41673
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:10.596470 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:10.675882 23541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:10.676105 23541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:10.676139 23541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:10.677317 23541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:10.677385 23541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:10.677429 23541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:10.678879 23541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:10.679147 23541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:10.679363 23541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:10.684207 23549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.684177 23547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:10.684190 23546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:10.684438 23541 server_base.cc:1034] running on GCE node
I20260927 03:18:10.684736 23541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:10.684978 23541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:10.686136 23541 hybrid_clock.cc:648] HybridClock initialized: now 1790479090686124 us; error 42 us; skew 500 ppm
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.688439 23541 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:10.688948 23541 webserver.cc:460] Webserver started at http://127.17.52.131:39577/ using document root <none> and password file <none>
I20260927 03:18:10.689177 23541 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:10.689242 23541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:10.689378 23541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:10.690464 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "c15e14ed8c14487ca456d5889c4b4fba"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "33319ddd516bbf0e2446dd188ae5185c"
server_key_iv: "35e493925844852b6d9ff5af51bfcdcb"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.690814 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "c15e14ed8c14487ca456d5889c4b4fba"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "33319ddd516bbf0e2446dd188ae5185c"
server_key_iv: "35e493925844852b6d9ff5af51bfcdcb"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.692049 23541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:10.692751 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.692926 23541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:10.693038 23541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "c15e14ed8c14487ca456d5889c4b4fba"
format_stamp: "Formatted at 2026-09-27 03:18:10 on dist-test-slave-7j43"
server_key: "33319ddd516bbf0e2446dd188ae5185c"
server_key_iv: "35e493925844852b6d9ff5af51bfcdcb"
server_key_version: "encryptionkey@0"
I20260927 03:18:10.693125 23541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:10.711164 23541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:10.712931 23541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:10.713078 23541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:10.713292 23541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:10.713588 23541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:10.713639 23541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.713680 23541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:10.713708 23541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:10.719270 23541 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:38485
I20260927 03:18:10.719377 23669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:38485 every 8 connection(s)
I20260927 03:18:10.719712 23541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:10.720513 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23541
I20260927 03:18:10.720619 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:10.720793 17618 external_mini_cluster.cc:1428] Setting key 191bb7f77b4195240e6cf732a0cf3276
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:10.729209 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.721101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:44629 (local address 127.17.52.190:33201)
0927 03:18:10.721177 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.721179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.721388 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:18:10.722315 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.722316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.722329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.722384 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.722994 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.723502 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.724495 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.724783 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.724866 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.726931 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.726932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.726933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.726940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.728078 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.728422 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.728422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.728423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.728469 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.728709 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.728710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.728710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.728820 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.728871 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.729092 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.729149 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:18:10.729393 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.721086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33201 (local address 127.17.52.131:44629)
0927 03:18:10.721250 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:10.721297 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:10.722195 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:10.722452 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:10.722456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:10.722711 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:10.722894 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.722897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.723591 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.723592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.724403 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:10.724405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.724875 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.724876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:10.724998 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.725505 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:10.725513 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:10.726850 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:10.728145 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.728150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.728152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.728359 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.728524 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:10.728524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:10.728525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:10.728653 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:10.728876 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:10.728880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:10.728987 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 03:18:10.729065 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:18:10.729173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260927 03:18:10.729858 23670 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33201
I20260927 03:18:10.729948 23670 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:10.730074 23670 heartbeater.cc:511] Master 127.17.52.190:33201 requested a full tablet report, sending...
I20260927 03:18:10.730388 23211 ts_manager.cc:194] Registered new tserver with Master: c15e14ed8c14487ca456d5889c4b4fba (127.17.52.131:38485)
I20260927 03:18:10.730705 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:44629
I20260927 03:18:10.732410 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/test-admin.keytab.
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.746556 17618 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 03:18:10.748101 23677 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:10.747389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.17.52.129:41731)
0927 03:18:10.747543 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.747548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.747557 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:10.747580 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.747581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.747593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.747658 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.748023 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36126: BlockingRecv error: recv got EOF from 127.0.0.1:36126 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260927 03:18:10.748225 17618 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.749236 17618 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.750123 17618 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:18:10.757654 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.750619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.17.52.129:41731)
0927 03:18:10.750670 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.750672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.750692 (+ 20us) server_negotiation.cc:407] Connection header received
0927 03:18:10.750743 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.750744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.750752 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.750800 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.751149 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.752002 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.752826 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.753110 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.753163 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.755079 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.755085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.755085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.755090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.756682 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.756969 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.756971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.756971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.757019 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.757161 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.757161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.757162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.757280 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.757376 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.757439 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.757494 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 03:18:10.759413 23677 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:10.758807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.17.52.129:41731)
0927 03:18:10.758873 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.758875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.758917 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:18:10.758966 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.758967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.758975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.759016 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.759364 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36144: BlockingRecv error: recv got EOF from 127.0.0.1:36144 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 03:18:10.759456 17618 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.760347 17618 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.761215 17618 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:18:10.767452 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.761683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.17.52.129:41731)
0927 03:18:10.761757 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.761758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.761802 (+ 44us) server_negotiation.cc:407] Connection header received
0927 03:18:10.761849 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.761849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.761857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.761900 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.762200 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.762646 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.763262 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.763444 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.763490 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.765299 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.765300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.765300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.765306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.766589 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.766862 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.766862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.766863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.766905 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.767043 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.767044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.767044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.767112 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.767169 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.767251 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.767310 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 03:18:10.768934 23677 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:10.768354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.17.52.129:41731)
0927 03:18:10.768412 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.768414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.768472 (+ 58us) server_negotiation.cc:407] Connection header received
0927 03:18:10.768529 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.768530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.768539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.768640 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.768892 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36164: BlockingRecv error: recv got EOF from 127.0.0.1:36164 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 03:18:10.769021 17618 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479066103559-17618-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.769901 17618 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:10.770676 17618 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:18:10.777089 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:10.771120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.17.52.129:41731)
0927 03:18:10.771171 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:10.771172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:10.771191 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:18:10.771235 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:10.771236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:10.771242 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:10.771293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:10.771593 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.772051 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.772777 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:10.772953 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:10.773002 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:10.774880 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:10.774881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:10.774881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:10.774887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:10.776145 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.776427 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.776428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.776428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.776471 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:10.776704 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:10.776704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:10.776704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:10.776780 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:10.776848 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:10.776925 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:18:10.776972 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:18:10.777683 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23273
I20260927 03:18:10.782611 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23407
I20260927 03:18:10.786557 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23541
I20260927 03:18:10.790563 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23179
2026-09-27T03:18:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1865 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23690](info): set up 2 sockets
Sep 27 03:18:10 dist-test-slave-7j43 krb5kdc[23690](info): commencing operation
krb5kdc: starting...
W20260927 03:18:11.821259 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:11 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab.
Entry for principal oryx/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab.
I20260927 03:18:11.932560 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:39093
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab
--principal=oryx/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:12.013262 23703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:12.013505 23703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:12.013545 23703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:12.014719 23703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:12.014793 23703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:12.014818 23703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:12.014839 23703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:12.014869 23703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:12.016932 23703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab
--principal=oryx/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:12.017215 23703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:12.017450 23703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:12.022322 23709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:12.022315 23711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:12.022317 23708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:12.022536 23703 server_base.cc:1034] running on GCE node
I20260927 03:18:12.022867 23703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:12.023126 23703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:12.024269 23703 hybrid_clock.cc:648] HybridClock initialized: now 1790479092024247 us; error 36 us; skew 500 ppm
Sep 27 03:18:12 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479092, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:12.026861 23703 init.cc:377] Logged in from keytab as oryx/127.17.52.190@KRBTEST.COM (short username oryx)
I20260927 03:18:12.027315 23703 webserver.cc:460] Webserver started at http://127.17.52.190:37569/ using document root <none> and password file <none>
I20260927 03:18:12.027542 23703 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:12.027617 23703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:12.027779 23703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:12.029126 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "e6e2eac3378c45d7a126d38846438bbc"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "46841b7b2a5d659872ad7d4f2a9adc6f"
server_key_iv: "19b4f73e94671685380dad68a8c4a61b"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.029549 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "e6e2eac3378c45d7a126d38846438bbc"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "46841b7b2a5d659872ad7d4f2a9adc6f"
server_key_iv: "19b4f73e94671685380dad68a8c4a61b"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.030867 23703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:18:12.031603 23718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:12.031774 23703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:12.031881 23703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "e6e2eac3378c45d7a126d38846438bbc"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "46841b7b2a5d659872ad7d4f2a9adc6f"
server_key_iv: "19b4f73e94671685380dad68a8c4a61b"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.031953 23703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:12.044790 23703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:12.047115 23703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:12.047283 23703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:12.051276 23703 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:39093
I20260927 03:18:12.051312 23770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:39093 every 8 connection(s)
I20260927 03:18:12.051663 23703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:12.052719 23771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:12.054879 23771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Bootstrap starting.
I20260927 03:18:12.055395 23771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:12.055614 23771 log.cc:824] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:12.056245 23771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: No bootstrap required, opened a new log
I20260927 03:18:12.056864 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23703
I20260927 03:18:12.056967 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:12.057157 17618 external_mini_cluster.cc:1428] Setting key 6cae315100774fb25887576500b0f645
I20260927 03:18:12.058396 23771 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } }
I20260927 03:18:12.058547 23771 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:12.058584 23771 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e2eac3378c45d7a126d38846438bbc, State: Initialized, Role: FOLLOWER
I20260927 03:18:12.058744 23771 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } }
I20260927 03:18:12.058835 23771 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:12.058871 23771 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:12.058912 23771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:12.059876 23771 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } }
I20260927 03:18:12.059963 23771 leader_election.cc:302] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e2eac3378c45d7a126d38846438bbc; no voters:
I20260927 03:18:12.060122 23771 leader_election.cc:288] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:12.060176 23776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:12.060317 23776 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 LEADER]: Becoming Leader. State: Replica: e6e2eac3378c45d7a126d38846438bbc, State: Running, Role: LEADER
I20260927 03:18:12.060396 23771 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:12.060405 23776 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } }
I20260927 03:18:12.060813 23777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } }
I20260927 03:18:12.060904 23777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:12.061141 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e2eac3378c45d7a126d38846438bbc. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } }
I20260927 03:18:12.061256 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:12.061538 23783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:12.062103 23783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:12.063586 23783 catalog_manager.cc:1347] Generated new cluster ID: 5a1b1a494d1745b9a0a4b457d099901d
I20260927 03:18:12.063656 23783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:18:12 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:12.069782 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:12.057680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.17.52.190:39093)
0927 03:18:12.057890 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:12.057899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:12.057929 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:18:12.058383 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:12.058390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:12.058440 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:12.058638 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:12.059393 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.060644 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.061979 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.062566 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.062682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:12.065802 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:12.065806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:12.065806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:12.065818 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:12.067725 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:12.068131 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:12.068133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:12.068134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:12.068620 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:12.068837 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:12.068837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:12.068838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:12.069162 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:12.069308 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:12.069443 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:18:12.069571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":122,"threads_started":1}
I20260927 03:18:12.076421 23783 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:12.076973 23783 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:12.084874 23783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Generated new TSK 0
I20260927 03:18:12.085089 23783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
I20260927 03:18:12.118949 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:12.206701 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:12.206907 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:12.206931 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:12.208105 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:12.208150 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:12.208176 23797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:12.209738 23797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:12.209987 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:12.210260 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:12.215687 23803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:12.215678 23802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:12.215678 23805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:12.215936 23797 server_base.cc:1034] running on GCE node
I20260927 03:18:12.216190 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:12.216449 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:12.217809 23797 hybrid_clock.cc:648] HybridClock initialized: now 1790479092217792 us; error 29 us; skew 500 ppm
Sep 27 03:18:12 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479092, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:12.220551 23797 init.cc:377] Logged in from keytab as oryx/127.17.52.129@KRBTEST.COM (short username oryx)
I20260927 03:18:12.221082 23797 webserver.cc:460] Webserver started at http://127.17.52.129:43181/ using document root <none> and password file <none>
I20260927 03:18:12.221346 23797 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:12.221436 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:12.221585 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:12.222679 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "b6e1ad773ea544168a227898f0a88bb5"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "89c97687a874242028c01349fa481c4a"
server_key_iv: "7fb1cb5d155b3d84b7f498711359fb2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.223080 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "b6e1ad773ea544168a227898f0a88bb5"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "89c97687a874242028c01349fa481c4a"
server_key_iv: "7fb1cb5d155b3d84b7f498711359fb2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.224532 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:18:12.225343 23812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:12.225517 23797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:12.225610 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "b6e1ad773ea544168a227898f0a88bb5"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "89c97687a874242028c01349fa481c4a"
server_key_iv: "7fb1cb5d155b3d84b7f498711359fb2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:12.225696 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:12.240734 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:12.242951 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:12.243134 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:12.243407 23797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:12.243736 23797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:12.243795 23797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:12.243839 23797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:12.243868 23797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:12.250370 23797 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:41095
I20260927 03:18:12.250412 23925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:41095 every 8 connection(s)
I20260927 03:18:12.250756 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:12.254431 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23797
I20260927 03:18:12.254516 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:12.254707 17618 external_mini_cluster.cc:1428] Setting key a3e35cad825e0e0a02ea3963d0623660
Sep 27 03:18:12 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479092, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:12.261560 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:12.251469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:57279 (local address 127.17.52.190:39093)
0927 03:18:12.251519 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:12.251537 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:12.251737 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:18:12.252292 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:12.252293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:12.252308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:12.252368 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:12.253424 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.253953 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.255211 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.255506 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.255594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:12.258256 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:12.258257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:12.258258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:12.258268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:12.259944 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:12.260421 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:12.260423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:12.260423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:12.260481 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:12.260784 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:12.260785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:12.260786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:12.260970 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:12.261064 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:12.261373 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 03:18:12.261469 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:12.261745 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:12.251443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:57279)
0927 03:18:12.251598 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:12.251645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:12.252181 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:12.252438 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:12.252443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:12.253110 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:12.253318 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:12.253323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.254063 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.254065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:12.255126 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:12.255128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:12.255604 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:12.255606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:12.255747 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:12.256308 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:12.256317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:12.258136 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:12.260034 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:12.260040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:12.260043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:12.260334 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:12.260553 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:12.260554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:12.260555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:12.260707 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:12.261065 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:12.261072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:12.261212 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:18:12.261322 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:18:12.261486 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260927 03:18:12.262243 23926 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:12.262326 23926 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:12.262558 23926 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.17.52.130@KRBTEST.COM; defaulting to no policy
I20260927 03:18:12.263170 23735 ts_manager.cc:194] Registered new tserver with Master: b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
I20260927 03:18:12.263918 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:57279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
W20260927 03:18:13.140105 17618 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.17.52.130: real 0.885s user 0.000s sys 0.003s
I20260927 03:18:13.140478 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:13.223078 23931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:13.223296 23931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:13.223320 23931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:13.224407 23931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:13.224443 23931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:13.224469 23931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:13.225948 23931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:13.226148 23931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:13.226316 23931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:13.231441 23936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:13.231474 23937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:13.231488 23931 server_base.cc:1034] running on GCE node
W20260927 03:18:13.231442 23939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:13.231797 23931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:13.232060 23931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:13.233218 23931 hybrid_clock.cc:648] HybridClock initialized: now 1790479093233200 us; error 44 us; skew 500 ppm
Sep 27 03:18:13 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479093, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:13.237684 23931 init.cc:377] Logged in from keytab as oryx/127.17.52.130@KRBTEST.COM (short username oryx)
I20260927 03:18:13.238152 23931 webserver.cc:460] Webserver started at http://127.17.52.130:35019/ using document root <none> and password file <none>
I20260927 03:18:13.238394 23931 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:13.238463 23931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:13.238624 23931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:13.239717 23931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
format_stamp: "Formatted at 2026-09-27 03:18:13 on dist-test-slave-7j43"
server_key: "fb8b3d2c65acaab30f344e1b01134a0d"
server_key_iv: "ce41ee24b24b7f295630543bfa2303fd"
server_key_version: "encryptionkey@0"
I20260927 03:18:13.240173 23931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
format_stamp: "Formatted at 2026-09-27 03:18:13 on dist-test-slave-7j43"
server_key: "fb8b3d2c65acaab30f344e1b01134a0d"
server_key_iv: "ce41ee24b24b7f295630543bfa2303fd"
server_key_version: "encryptionkey@0"
I20260927 03:18:13.241658 23931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:13.242481 23946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:13.242686 23931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:13.242811 23931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
format_stamp: "Formatted at 2026-09-27 03:18:13 on dist-test-slave-7j43"
server_key: "fb8b3d2c65acaab30f344e1b01134a0d"
server_key_iv: "ce41ee24b24b7f295630543bfa2303fd"
server_key_version: "encryptionkey@0"
I20260927 03:18:13.242888 23931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:13.262701 23931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:13.264731 23931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:13.264870 23926 heartbeater.cc:503] Master 127.17.52.190:39093 was elected leader, sending a full tablet report...
I20260927 03:18:13.264907 23931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:13.265174 23931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:13.265542 23931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:13.265589 23931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:13.265618 23931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:13.265636 23931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:13.271543 23931 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:34295
I20260927 03:18:13.271589 24059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:34295 every 8 connection(s)
I20260927 03:18:13.271903 23931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:13.275578 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 23931
I20260927 03:18:13.275661 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:13.275874 17618 external_mini_cluster.cc:1428] Setting key d1a117064f868099251e64312b396027
Sep 27 03:18:13 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479093, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:13.282091 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:13.272641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:51311)
0927 03:18:13.272786 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:13.272829 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:13.273371 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:13.273594 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:13.273598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:13.274210 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:13.274417 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:13.274422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:13.275099 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:13.275100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:13.276102 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:13.276105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:13.276553 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:13.276554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:13.276709 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:13.277252 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:13.277272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:13.278677 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:13.280716 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:13.280734 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:13.280736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:13.280953 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:13.281175 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:13.281175 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:13.281176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:13.281276 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:13.281581 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:13.281585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:13.281700 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 03:18:13.281799 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:18:13.281918 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260927 03:18:13.282078 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:13.272629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:51311 (local address 127.17.52.190:39093)
0927 03:18:13.272763 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:13.272765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:13.272911 (+ 146us) server_negotiation.cc:407] Connection header received
0927 03:18:13.273457 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:13.273457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:13.273468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:13.273535 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:13.274501 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:13.275020 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:13.276188 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:13.276468 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:13.276531 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:13.278771 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:13.278772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:13.278773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:13.278781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:13.280629 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:13.281032 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:13.281034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:13.281034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:13.281104 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:13.281343 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:13.281344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:13.281345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:13.281510 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:13.281590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:13.281853 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 03:18:13.281959 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260927 03:18:13.282593 24060 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:13.282712 24060 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:13.282907 24060 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
I20260927 03:18:13.283354 23735 ts_manager.cc:194] Registered new tserver with Master: b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
I20260927 03:18:13.283874 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:51311
WARNING: no policy specified for oryx/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
W20260927 03:18:14.007387 17618 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.17.52.131: real 0.731s user 0.000s sys 0.003s
I20260927 03:18:14.007746 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:14.092693 24066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:14.092885 24066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:14.092908 24066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:14.094244 24066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:14.094295 24066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:14.094365 24066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:14.095912 24066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:14.096150 24066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:14.096379 24066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.101173 24072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.101217 24074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.101166 24071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.101188 24066 server_base.cc:1034] running on GCE node
I20260927 03:18:14.101507 24066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.101789 24066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.102943 24066 hybrid_clock.cc:648] HybridClock initialized: now 1790479094102929 us; error 32 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.105417 24066 init.cc:377] Logged in from keytab as oryx/127.17.52.131@KRBTEST.COM (short username oryx)
I20260927 03:18:14.105772 24066 webserver.cc:460] Webserver started at http://127.17.52.131:38557/ using document root <none> and password file <none>
I20260927 03:18:14.105958 24066 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.105996 24066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.106081 24066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:14.107009 24066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "f7cd6500ef184de096d1f8843bc3e5e1"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "af326f89e826ba44eb860fb1233eca54"
server_key_iv: "a0731b2ba316545bcd029afcd9e69297"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.107287 24066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "f7cd6500ef184de096d1f8843bc3e5e1"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "af326f89e826ba44eb860fb1233eca54"
server_key_iv: "a0731b2ba316545bcd029afcd9e69297"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.108466 24066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:18:14.109215 24081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.109388 24066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:14.109485 24066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "f7cd6500ef184de096d1f8843bc3e5e1"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "af326f89e826ba44eb860fb1233eca54"
server_key_iv: "a0731b2ba316545bcd029afcd9e69297"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.109557 24066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.116812 24066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.118930 24066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:14.119081 24066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.119295 24066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:14.119572 24066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:14.119604 24066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.119624 24066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:14.119669 24066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.125278 24066 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42637
I20260927 03:18:14.125353 24194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42637 every 8 connection(s)
I20260927 03:18:14.125641 24066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:14.132513 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24066
I20260927 03:18:14.132639 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:14.132830 17618 external_mini_cluster.cc:1428] Setting key 851845a3c20c906ec1ac259b0914e07e
I20260927 03:18:14.134841 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.126348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:60803 (local address 127.17.52.190:39093)
0927 03:18:14.126493 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.126496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.126602 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:18:14.127049 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.127050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.127061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.127122 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.128007 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.128604 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.129653 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.129852 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.129905 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.131876 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.131878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.131878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.131884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.133613 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.133973 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.133974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.133974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.134026 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.134225 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.134226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.134226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.134334 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.134385 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.134652 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.134768 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 03:18:14.134971 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.126331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:60803)
0927 03:18:14.126485 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.126517 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.126964 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.127190 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.127193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.127731 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.127924 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.127928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.128680 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.128684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.129577 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.129580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.129919 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.129920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.130020 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.130500 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.130508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.131796 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.133690 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.133695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.133696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.133914 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.134082 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.134083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.134083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.134173 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.134388 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.134392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.134545 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.134625 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.134769 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260927 03:18:14.135406 24195 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:14.135476 24195 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.135659 24195 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
I20260927 03:18:14.136049 23735 ts_manager.cc:194] Registered new tserver with Master: f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637)
I20260927 03:18:14.136410 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:60803
I20260927 03:18:14.144372 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.52.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.189.keytab.
Entry for principal oryx/127.17.52.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.189.keytab.
I20260927 03:18:14.167186 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:40151
--webserver_interface=127.17.52.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.189.keytab
--principal=oryx/127.17.52.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:39093,127.17.52.189:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:14.247462 24201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:14.247666 24201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:14.247687 24201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:14.248857 24201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:14.248898 24201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:14.248911 24201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:14.248924 24201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:14.248936 24201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:14.250494 24201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:39093,127.17.52.189:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.189.keytab
--principal=oryx/127.17.52.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.52.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
W20260927 03:18:14.250630 24201 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.52.190:39093,127.17.52.189:40151'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 03:18:14.250797 24201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:14.251016 24201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.255683 24207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.255687 24209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.255687 24206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.256072 24201 server_base.cc:1034] running on GCE node
I20260927 03:18:14.256295 24201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.256623 24201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.257778 24201 hybrid_clock.cc:648] HybridClock initialized: now 1790479094257769 us; error 26 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.260318 24201 init.cc:377] Logged in from keytab as oryx/127.17.52.189@KRBTEST.COM (short username oryx)
I20260927 03:18:14.260789 24201 webserver.cc:460] Webserver started at http://127.17.52.189:42887/ using document root <none> and password file <none>
I20260927 03:18:14.261008 24201 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.261060 24201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.261188 24201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:14.262272 24201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/instance:
uuid: "4abc28b0becc4465a9339e851140f6a2"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "775cfbbdc05ae706c66a61f87fec4983"
server_key_iv: "3f52fa2543c67483d23ce21f1ee3d35b"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.262625 24201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal/instance:
uuid: "4abc28b0becc4465a9339e851140f6a2"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "775cfbbdc05ae706c66a61f87fec4983"
server_key_iv: "3f52fa2543c67483d23ce21f1ee3d35b"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.263913 24201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:14.264642 24216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.264847 24201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:14.264946 24201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "4abc28b0becc4465a9339e851140f6a2"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "775cfbbdc05ae706c66a61f87fec4983"
server_key_iv: "3f52fa2543c67483d23ce21f1ee3d35b"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.265026 24201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.278029 24201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.280390 24201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:14.280557 24201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.284852 24201 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.189:40151
I20260927 03:18:14.284883 24268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.189:40151 every 8 connection(s)
I20260927 03:18:14.284870 24060 heartbeater.cc:503] Master 127.17.52.190:39093 was elected leader, sending a full tablet report...
I20260927 03:18:14.285298 24201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
I20260927 03:18:14.286279 24269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:14.287751 24269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:18:14.292146 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24201
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.189@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:14.297545 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.288362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.17.52.190:39093)
0927 03:18:14.288445 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.288447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.288650 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:18:14.289139 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.289140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.289150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.289207 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.289786 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.290296 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.291298 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.291501 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.291559 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.294277 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.294278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.294278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.294285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.295894 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.296386 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.296387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.296387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.296444 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.296742 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.296743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.296743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.296970 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.297053 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.297340 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.297454 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:18:14.297715 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.288322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42748)
0927 03:18:14.288501 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.288536 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.289017 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.289259 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.289262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.289511 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.289699 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.289703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.290384 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.290384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.291208 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.291210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.291578 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.291579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.291693 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.292154 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.292174 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.294171 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.295980 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.295986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.295987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.296309 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.296514 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.296514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.296515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.296672 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.297058 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.297064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.297203 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.297307 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.297472 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260927 03:18:14.298424 24269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
I20260927 03:18:14.302430 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.292655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.17.52.189:40151)
0927 03:18:14.292813 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.292818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.292997 (+ 179us) server_negotiation.cc:407] Connection header received
0927 03:18:14.293021 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.293022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.293040 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.293102 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.293828 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.294955 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.296054 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.296516 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.296609 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.299294 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.299300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.299301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.299307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.301001 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.301435 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.301438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.301438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.301498 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.301700 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.301701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.301701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.301837 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.302007 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.302079 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.302176 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.189@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
I20260927 03:18:14.305335 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.298716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:39976)
0927 03:18:14.298758 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.298767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.298802 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.299060 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.299061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.299202 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.299264 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.299266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.300028 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.300029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.300866 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.300869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.301303 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.301303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.301355 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.301752 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.301756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.303018 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.304362 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.304362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.304363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.304521 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.304737 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.304738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.304738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.304790 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.305028 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.305029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.305065 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.305130 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.305224 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:18:14.305433 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.298799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.17.52.189:40151)
0927 03:18:14.298945 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.298946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.298953 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:14.298969 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.298969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.298977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.299058 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.299337 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.299954 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.300944 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.301226 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.301299 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.303097 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.303098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.303098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.303104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.304270 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.304614 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.304615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.304615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.304672 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.304861 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.304862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.304863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.304964 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.305037 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.305166 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.305273 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 03:18:14.306823 24269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Bootstrap starting.
I20260927 03:18:14.307389 24269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:14.307607 24269 log.cc:824] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:14.308274 24269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: No bootstrap required, opened a new log
I20260927 03:18:14.309681 24269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } }
I20260927 03:18:14.309800 24269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:14.309823 24269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abc28b0becc4465a9339e851140f6a2, State: Initialized, Role: FOLLOWER
I20260927 03:18:14.309940 24269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } }
I20260927 03:18:14.310177 24276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } }
I20260927 03:18:14.310245 24269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:14.310257 24276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:18:14.311853 24287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:18:14.311931 24287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:18:14.313370 24201 master_runner.cc:428] Detected that this master 4abc28b0becc4465a9339e851140f6a2 is joining an existing cluster
I20260927 03:18:14.313422 24201 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.52.189:40151
I20260927 03:18:14.321187 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.314432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.17.52.190:39093)
0927 03:18:14.314477 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.314479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.314597 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:18:14.314724 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.314725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.314738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.314803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.315237 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.315975 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.316855 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.317097 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.317159 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.318301 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.318303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.318303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.318310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.319845 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320324 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.320325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.320326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.320381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320599 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.320600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.320600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.320722 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.320813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.321009 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.321096 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:18:14.321192 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.314346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42752)
0927 03:18:14.314534 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.314544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.314630 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.314810 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.314811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.315045 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.315128 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.315130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.316065 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.316066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.316753 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.316755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.317170 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.317171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.317235 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.317785 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.317791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.318208 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.319947 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.319948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.319949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.320246 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.320455 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.320456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.320513 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.320806 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.320807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.320849 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.320969 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.321073 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":48,"thread_start_us":104,"threads_started":1}
I20260927 03:18:14.321820 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.314619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.17.52.189:40151)
0927 03:18:14.314653 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.314655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.314754 (+ 99us) server_negotiation.cc:407] Connection header received
0927 03:18:14.314846 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.314847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.314855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.314878 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.315198 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.315943 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.317085 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.317328 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.317412 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.318376 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.318377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.318377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.318383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.320293 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320691 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.320692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.320692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.320742 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320931 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.320932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.320933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.321048 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.321105 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.321620 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.321740 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":18}
I20260927 03:18:14.321820 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.314583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:39980)
0927 03:18:14.314700 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.314709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.314783 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.314924 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.314924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.315074 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.315126 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.315127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.316030 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.316031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.317001 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.317004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.317414 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.317414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.317474 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.317945 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.317950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.318298 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.320379 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.320380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.320605 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.320809 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.320809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.320810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.320866 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.321127 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.321128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.321500 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.321594 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.321690 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":27,"thread_start_us":63,"threads_started":1}
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479092, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.190@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
I20260927 03:18:14.330605 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.323694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.17.52.189:40151)
0927 03:18:14.323733 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.323735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.323898 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:18:14.323950 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.323951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.323962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.324029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.324462 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.324979 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.325752 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.325955 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.326060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.328147 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.328148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.328148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.328154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.329528 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.329946 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.329946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.329946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.329990 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.330172 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.330173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.330173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.330257 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.330305 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.330500 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.330551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:18:14.330756 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.323645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:39984)
0927 03:18:14.323786 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.323826 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.323897 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.324032 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.324034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.324278 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.324373 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.324375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.325059 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.325061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.325651 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.325652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.326046 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.326047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.326134 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.326548 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.326552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.328070 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.329597 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.329604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.329606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.329868 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.330048 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.330049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.330049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.330113 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.330316 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.330323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.330380 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.330478 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.330591 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260927 03:18:14.331259 23735 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.52.189:40151: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4abc28b0becc4465a9339e851140f6a2"
member_type: NON_VOTER
last_known_addr {
host: "127.17.52.189"
port: 40151
}
attrs {
promote: true
}
}
dest_uuid: "e6e2eac3378c45d7a126d38846438bbc"
cas_config_opid_index: -1
I20260927 03:18:14.331573 23735 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } }
I20260927 03:18:14.332160 24297 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.332408 24243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:14.333315 24297 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.52.189:40151
I20260927 03:18:14.333325 24243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 03:18:14.333539 24298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.333645 24298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:14.333748 24201 master.cc:537] Master@127.17.52.189:40151 shutting down...
I20260927 03:18:14.333742 24297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:18:14.334585 23721 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 03:18:14.336143 24201 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:14.336226 24201 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:14.336275 24201 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: stopping tablet replica
I20260927 03:18:14.340968 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.340834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:39986)
0927 03:18:14.340890 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.340935 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:40151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:18:14.341089 23721 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc -> Peer 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189:40151): Couldn't send request to peer 4abc28b0becc4465a9339e851140f6a2. Status: Network error: Client connection negotiation failed: client connection to 127.17.52.189:40151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:18:14.349081 24201 master.cc:552] Master@127.17.52.189:40151 shutdown complete.
I20260927 03:18:14.349123 24201 master_runner.cc:305] Clearing existing system tablet
I20260927 03:18:14.349273 24201 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.349325 24201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.349970 24201 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:14.350415 24302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.350548 24201 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.350601 24201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.350627 24201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "4abc28b0becc4465a9339e851140f6a2"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "775cfbbdc05ae706c66a61f87fec4983"
server_key_iv: "3f52fa2543c67483d23ce21f1ee3d35b"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.351239 24201 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:14.352423 24201 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:18:14.352475 24201 log.cc:1196] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 03:18:14.352725 24201 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Deleting consensus metadata
I20260927 03:18:14.352864 24201 master_runner.cc:315] Copying system tablet from 127.17.52.190:39093
I20260927 03:18:14.353477 24201 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.52.190:39093
I20260927 03:18:14.359619 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.353886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.17.52.190:39093)
0927 03:18:14.353931 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.353933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.353979 (+ 46us) server_negotiation.cc:407] Connection header received
0927 03:18:14.354050 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.354051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.354060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.354104 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.354449 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.354940 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.355970 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.356142 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.356216 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.357212 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.357213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.357213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.357221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.358483 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.358848 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.358849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.358849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.358903 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.359110 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.359111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.359112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.359225 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.359307 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.359465 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.359530 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:14.359637 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.353795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42764)
0927 03:18:14.353914 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.353924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.353989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.354119 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.354119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.354312 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.354380 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.354383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.355015 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.355016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.355890 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.355893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.356229 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.356230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.356307 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.356833 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.356837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.357107 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.358586 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.358587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.358588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.358759 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.358983 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.358984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.358984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.359047 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.359298 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.359299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.359343 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.359428 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.359530 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260927 03:18:14.359894 23755 tablet_copy_service.cc:140] P e6e2eac3378c45d7a126d38846438bbc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4abc28b0becc4465a9339e851140f6a2 ({username='oryx', principal='oryx/127.17.52.189@KRBTEST.COM'} at 127.0.0.1:42764)
I20260927 03:18:14.359997 23755 tablet_copy_service.cc:161] P e6e2eac3378c45d7a126d38846438bbc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4abc28b0becc4465a9339e851140f6a2 at {username='oryx', principal='oryx/127.17.52.189@KRBTEST.COM'} at 127.0.0.1:42764: session id = 4abc28b0becc4465a9339e851140f6a2-00000000000000000000000000000000
I20260927 03:18:14.360735 23755 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:18:14.361317 24201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:18:14.362300 24201 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: tablet copy: Starting download of 0 data blocks...
I20260927 03:18:14.362458 24201 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: tablet copy: Starting download of 1 WAL segments...
I20260927 03:18:14.363372 24201 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:18:14.364091 23755 tablet_copy_service.cc:342] P e6e2eac3378c45d7a126d38846438bbc: Request end of tablet copy session 4abc28b0becc4465a9339e851140f6a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.52.189@KRBTEST.COM'} at 127.0.0.1:42764
I20260927 03:18:14.364184 23755 tablet_copy_service.cc:433] P e6e2eac3378c45d7a126d38846438bbc: ending tablet copy session 4abc28b0becc4465a9339e851140f6a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4abc28b0becc4465a9339e851140f6a2
I20260927 03:18:14.366652 24201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.367794 24309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.367797 24310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.369515 24201 server_base.cc:1034] running on GCE node
W20260927 03:18:14.369580 24312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.369705 24201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.369848 24201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.370988 24201 hybrid_clock.cc:648] HybridClock initialized: now 1790479094370978 us; error 21 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.372275 24201 init.cc:377] Logged in from keytab as oryx/127.17.52.189@KRBTEST.COM (short username oryx)
I20260927 03:18:14.372548 24201 webserver.cc:460] Webserver started at http://127.17.52.189:44681/ using document root <none> and password file <none>
I20260927 03:18:14.372692 24201 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.372741 24201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.373339 24201 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:14.373831 24319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.373956 24201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.374004 24201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "4abc28b0becc4465a9339e851140f6a2"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "775cfbbdc05ae706c66a61f87fec4983"
server_key_iv: "3f52fa2543c67483d23ce21f1ee3d35b"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.374083 24201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.380053 24201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.381593 24201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.384585 24201 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.189:40151
I20260927 03:18:14.384977 24371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.189:40151 every 8 connection(s)
I20260927 03:18:14.385003 24201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
I20260927 03:18:14.385686 24372 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:18:14.386103 24372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Bootstrap starting.
I20260927 03:18:14.388238 24372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:18:14.388468 24372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: Bootstrap complete.
I20260927 03:18:14.388700 24372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } }
I20260927 03:18:14.388773 24372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4abc28b0becc4465a9339e851140f6a2, State: Initialized, Role: LEARNER
I20260927 03:18:14.388799 24372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } }
I20260927 03:18:14.388968 24372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:14.388990 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.389083 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:18:14.390331 24386 catalog_manager.cc:1259] Loaded cluster ID: 5a1b1a494d1745b9a0a4b457d099901d
I20260927 03:18:14.390404 24386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: loading cluster ID for follower catalog manager: success
I20260927 03:18:14.391340 24386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: acquiring CA information for follower catalog manager: success
I20260927 03:18:14.391616 24386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:18:14.410909 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.403912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.17.52.189:40151)
0927 03:18:14.404026 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.404028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.404036 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:14.404057 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.404058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.404066 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.404128 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.404870 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.405986 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.406713 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.406936 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.407010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.408207 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.408208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.408209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.408216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.409743 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.410104 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.410105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.410105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.410161 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.410393 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.410394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.410394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.410503 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.410605 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.410708 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.410790 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.17.52.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.188.keytab.
Entry for principal oryx/127.17.52.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.188.keytab.
I20260927 03:18:14.433233 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:46271
--webserver_interface=127.17.52.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.188.keytab
--principal=oryx/127.17.52.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:14.513512 24390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:14.513707 24390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:14.513728 24390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:14.514803 24390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:14.514838 24390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:14.514853 24390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:14.514865 24390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:14.514878 24390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:14.516252 24390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.188.keytab
--principal=oryx/127.17.52.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.52.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:14.516443 24390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:14.516615 24390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.521195 24396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.521200 24395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.521214 24398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.521529 24390 server_base.cc:1034] running on GCE node
I20260927 03:18:14.521699 24390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.521972 24390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.523130 24390 hybrid_clock.cc:648] HybridClock initialized: now 1790479094523112 us; error 30 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.525965 24390 init.cc:377] Logged in from keytab as oryx/127.17.52.188@KRBTEST.COM (short username oryx)
I20260927 03:18:14.526427 24390 webserver.cc:460] Webserver started at http://127.17.52.188:45275/ using document root <none> and password file <none>
I20260927 03:18:14.526647 24390 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.526712 24390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.526866 24390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:14.527926 24390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/instance:
uuid: "01a07a3a10bf43359a40cf357b1f4718"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "b79c07f03e76897cd7c5d0bba8201e66"
server_key_iv: "eb0e5d8e4f857058759d4d663691a60d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.528281 24390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal/instance:
uuid: "01a07a3a10bf43359a40cf357b1f4718"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "b79c07f03e76897cd7c5d0bba8201e66"
server_key_iv: "eb0e5d8e4f857058759d4d663691a60d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.529652 24390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:14.530391 24405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.530572 24390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:14.530674 24390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "01a07a3a10bf43359a40cf357b1f4718"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "b79c07f03e76897cd7c5d0bba8201e66"
server_key_iv: "eb0e5d8e4f857058759d4d663691a60d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.530752 24390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.543099 24390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.545331 24390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:14.545475 24390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.549412 24390 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.188:46271
I20260927 03:18:14.549472 24457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.188:46271 every 8 connection(s)
I20260927 03:18:14.549823 24390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
I20260927 03:18:14.550730 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:14.552242 24458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.188@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:14.557664 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24390
I20260927 03:18:14.561014 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.553025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.17.52.190:39093)
0927 03:18:14.553071 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.553074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.553206 (+ 132us) server_negotiation.cc:407] Connection header received
0927 03:18:14.553683 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.553684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.553696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.553746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.554304 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.554801 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.555772 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.555978 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.556063 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.558085 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.558087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.558088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.558097 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.559542 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.559972 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.559972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.559973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.560022 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.560302 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.560303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.560303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.560453 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.560523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.560845 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.560919 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:14.561131 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.552895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42768)
0927 03:18:14.553059 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.553103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.553557 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.553814 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.553817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.554036 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.554212 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.554216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.554898 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.554899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.555682 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.555684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.556083 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.556084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.556208 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.556704 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.556711 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.557996 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.559620 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.559625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.559627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.559901 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.560084 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.560098 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.560098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.560237 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.560519 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.560524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.560687 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.560781 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.560938 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260927 03:18:14.561607 24458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.188@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
I20260927 03:18:14.566548 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.558179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.17.52.188:46271)
0927 03:18:14.558305 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.558311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.558334 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:18:14.558355 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.558357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.558369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.558447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.559194 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.560097 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.561134 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.561587 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.561650 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.563784 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.563790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.563791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.563797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.565229 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.565633 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.565636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.565636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.565694 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.565893 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.565894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.565894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.566062 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.566195 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.566266 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.566418 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260927 03:18:14.567948 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.561851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:39994)
0927 03:18:14.561925 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.561934 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.561984 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.562128 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.562129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.562263 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.562300 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.562301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.562951 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.562952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.563512 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.563514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.563937 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.563938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.563994 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.564487 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.564493 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.565621 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.567194 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.567195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.567195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.567339 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.567535 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.567536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.567536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.567581 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.567757 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.567758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.567788 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.567836 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.567896 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 03:18:14.567965 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.561925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.17.52.189:40151)
0927 03:18:14.561985 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.561987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.561995 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:14.562042 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.562043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.562054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.562126 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.562374 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.562840 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.563598 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.563860 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.563933 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.565678 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.565680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.565680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.565686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.567129 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.567396 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.567397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.567397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.567453 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.567622 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.567623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.567623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.567712 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.567753 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.567853 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.567910 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:18:14.568270 24458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.188@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
I20260927 03:18:14.573294 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.568476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.0.0.1:35146)
0927 03:18:14.568520 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.568531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.568614 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.568731 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.568731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.568827 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.568851 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.568852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.569404 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.569404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.569838 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.569839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.570129 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.570130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.570162 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.570422 (+ 260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.570426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.571398 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.572562 (+ 1164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.572563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.572563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.572756 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.572912 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.572913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.572913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.572957 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.573110 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.573111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.573139 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.573186 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.573245 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:18:14.573351 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.568612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.17.52.188:46271)
0927 03:18:14.568643 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.568645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.568653 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:14.568670 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.568670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.568679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.568729 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.568905 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.569339 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.569899 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.570072 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.570121 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.571467 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.571468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.571468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.571473 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.572499 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.572813 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.572813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.572814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.572858 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.572992 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.572992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.572992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.573065 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.573105 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.573198 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.573287 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":15}
I20260927 03:18:14.574513 24458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Bootstrap starting.
I20260927 03:18:14.575282 24458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:14.575493 24458 log.cc:824] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:14.576166 24458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: No bootstrap required, opened a new log
I20260927 03:18:14.577741 24458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } }
I20260927 03:18:14.577852 24458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:14.577873 24458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a07a3a10bf43359a40cf357b1f4718, State: Initialized, Role: FOLLOWER
I20260927 03:18:14.577955 24458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } }
I20260927 03:18:14.578173 24463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } } }
I20260927 03:18:14.578231 24458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:14.578260 24463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:18:14.579699 24474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:18:14.579804 24474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:18:14.581815 24390 master_runner.cc:428] Detected that this master 01a07a3a10bf43359a40cf357b1f4718 is joining an existing cluster
I20260927 03:18:14.581883 24390 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.52.188:46271
I20260927 03:18:14.589721 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.583204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.17.52.188:46271)
0927 03:18:14.583285 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.583287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.583445 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:18:14.583529 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.583529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.583536 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.583561 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.583994 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584704 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.585721 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.585988 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586025 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.587160 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.587160 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.587161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.587166 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.588534 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.589005 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.589006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.589006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.589052 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.589267 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.589268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.589268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.589382 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.589433 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.589597 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.589666 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":65}
I20260927 03:18:14.589830 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.583121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.0.0.1:35156)
0927 03:18:14.583401 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.583409 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.583470 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.583612 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.583612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.583861 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.583916 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.583918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584790 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.584790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.585632 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.585635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.586090 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.586148 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.586668 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.586673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.587079 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.588670 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.588672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.588672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.588936 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.589117 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.589117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.589118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.589180 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.589448 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.589448 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.589500 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.589581 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.589730 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":60,"thread_start_us":52,"threads_started":1}
I20260927 03:18:14.605257 24463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:14.605401 24463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } }
I20260927 03:18:14.605670 24463 leader_election.cc:288] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093), 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189:40151)
I20260927 03:18:14.605871 23745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a07a3a10bf43359a40cf357b1f4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6e2eac3378c45d7a126d38846438bbc" is_pre_election: true
I20260927 03:18:14.605913 24346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a07a3a10bf43359a40cf357b1f4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abc28b0becc4465a9339e851140f6a2" is_pre_election: true
I20260927 03:18:14.606045 24346 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Handling vote request from an unknown peer 01a07a3a10bf43359a40cf357b1f4718
I20260927 03:18:14.606096 24346 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 01a07a3a10bf43359a40cf357b1f4718 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 03:18:14.606335 24408 leader_election.cc:302] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01a07a3a10bf43359a40cf357b1f4718; no voters: 4abc28b0becc4465a9339e851140f6a2, e6e2eac3378c45d7a126d38846438bbc
I20260927 03:18:14.606465 24463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:14.607183 24463 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 03:18:14.612988 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.582784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42772)
0927 03:18:14.583096 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.583122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.583239 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.583432 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.583433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.583734 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.583825 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.583826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584779 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.584780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.585866 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.585867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.586351 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.586403 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.609147 (+ 22744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.609153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.609831 (+ 678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.611671 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.611672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.611673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.612021 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.612217 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.612218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.612272 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.612603 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.612604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.612655 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.612761 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.612876 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":52,"thread_start_us":73,"threads_started":1}
I20260927 03:18:14.612988 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.582892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.17.52.190:39093)
0927 03:18:14.582932 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.582934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.583180 (+ 246us) server_negotiation.cc:407] Connection header received
0927 03:18:14.583324 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.583325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.583336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.583367 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.583913 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584682 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.585942 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.586245 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586285 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.609932 (+ 23647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.609934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.609934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.609943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.611584 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612093 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.612094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.612095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.612153 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612356 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.612357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.612357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.612511 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.612620 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.612791 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.612883 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:18:14.613531 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.583247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.17.52.189:40151)
0927 03:18:14.583316 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.583317 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.583325 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:14.583369 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.583370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.583378 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.583428 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.583913 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584708 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.585813 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.586091 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586126 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.609953 (+ 23827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.609954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.609955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.609965 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.611589 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612239 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.612240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.612241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.612299 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612499 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.612501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.612501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.612678 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.612765 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.612971 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.613429 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 03:18:14.613873 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.582975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:40006)
0927 03:18:14.583260 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.583279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.583322 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.583484 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.583485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.583729 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.583822 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.583823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.584801 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.584802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.585721 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.585723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.586195 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.586196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.586258 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.609359 (+ 23101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.609364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.609829 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.611697 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.611698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.611698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.612021 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.612371 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.612372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.612373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.612431 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.612809 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.612810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.612853 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.613666 (+ 813us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.613772 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":48,"thread_start_us":61,"threads_started":1}
I20260927 03:18:14.618045 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23797
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479092, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.190@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
I20260927 03:18:14.622447 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.615381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.17.52.188:46271)
0927 03:18:14.615425 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.615427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.615434 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:14.615501 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.615501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.615509 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.615570 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.615848 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.616308 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.617051 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.617285 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.617345 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.619586 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.619587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.619588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.619595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.621339 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.621699 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.621700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.621701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.621752 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.621957 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.621958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.621958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.622065 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.622140 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.622294 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.622363 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:18:14.622454 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.615327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.0.0.1:35172)
0927 03:18:14.615378 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.615389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.615432 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.615590 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.615591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.615732 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.615773 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.615774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.616385 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.616386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.616972 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.616973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.617415 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.617416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.617482 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.617884 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.617889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.619495 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.621425 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.621426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.621426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.621627 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.621829 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.621830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.621830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.621887 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.622133 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.622134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.622177 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.622265 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.622364 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:18:14.622846 23735 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.52.188:46271: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718"
member_type: NON_VOTER
last_known_addr {
host: "127.17.52.188"
port: 46271
}
attrs {
promote: true
}
}
dest_uuid: "e6e2eac3378c45d7a126d38846438bbc"
cas_config_opid_index: 5
I20260927 03:18:14.623075 23735 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
I20260927 03:18:14.623430 24298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.624253 24298 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.52.188:46271
I20260927 03:18:14.624325 24298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.624392 24298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:14.624696 24432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:18:14.624913 24297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:14.625813 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:41095
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:39093
--webserver_port=43181
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:18:14.627084 24432 raft_consensus.cc:858] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
W20260927 03:18:14.627175 24432 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
I20260927 03:18:14.627653 24432 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.628923 24390 master.cc:537] Master@127.17.52.188:46271 shutting down...
I20260927 03:18:14.629074 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.623503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:40018)
0927 03:18:14.625576 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.625586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.625626 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.625831 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.625832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.625833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:14.625871 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.625873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.626770 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.626771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.628007 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.628029 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.628826 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.628827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.628893 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.628918 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.628976 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.629011 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059}
I20260927 03:18:14.629347 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.623580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.17.52.189:40151)
0927 03:18:14.623686 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.623688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.625679 (+ 1991us) server_negotiation.cc:407] Connection header received
0927 03:18:14.625702 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.625703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.625725 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.625779 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:14.625940 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.626679 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.628122 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.628742 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.628815 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.628885 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.629213 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.629253 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 03:18:14.629516 24297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.629585 24297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:14.629772 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e2eac3378c45d7a126d38846438bbc. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.629884 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:18:14.630005 24487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.630064 24487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:18:14.630208 24346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260927 03:18:14.630226 23721 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc -> Peer 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188:46271): Couldn't send request to peer 01a07a3a10bf43359a40cf357b1f4718. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:18:14.630537 24298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 03:18:14.631121 24463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } }
I20260927 03:18:14.631192 24463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:18:14.631069 24346 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.632014 24483 mvcc.cc:205] Tried to move back new op lower bound from 7333802361177526272 to 7333802361079869440. Current Snapshot: MvccSnapshot[applied={T|T < 7333802361089261568}]
I20260927 03:18:14.632258 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e2eac3378c45d7a126d38846438bbc. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.632357 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:18:14.632405 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: NON_VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: true } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.632439 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:18:14.633440 24298 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: attempting to promote NON_VOTER 4abc28b0becc4465a9339e851140f6a2 to VOTER
I20260927 03:18:14.633627 24298 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
I20260927 03:18:14.633996 24390 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 03:18:14.634112 24390 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 03:18:14.634164 24390 tablet_replica.cc:330] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: stopping tablet replica
I20260927 03:18:14.634399 24346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 LEARNER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 03:18:14.634605 24297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 03:18:14.634680 23721 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc -> Peer 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188:46271): Couldn't send request to peer 01a07a3a10bf43359a40cf357b1f4718. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:18:14.635171 24297 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.636142 24297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.636236 24297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:14.636291 24297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.636351 24297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:14.636813 24346 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.637465 24491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.637604 24491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:18:14.647413 24390 master.cc:552] Master@127.17.52.188:46271 shutdown complete.
I20260927 03:18:14.647464 24390 master_runner.cc:305] Clearing existing system tablet
I20260927 03:18:14.647593 24390 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.647635 24390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.648348 24390 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:14.648902 24495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.649041 24390 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.649106 24390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.649133 24390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "01a07a3a10bf43359a40cf357b1f4718"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "b79c07f03e76897cd7c5d0bba8201e66"
server_key_iv: "eb0e5d8e4f857058759d4d663691a60d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.649833 24390 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:14.651222 24390 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:18:14.651302 24390 log.cc:1196] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 03:18:14.651546 24390 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Deleting consensus metadata
I20260927 03:18:14.651672 24390 master_runner.cc:315] Copying system tablet from 127.17.52.190:39093
I20260927 03:18:14.652297 24390 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: tablet copy: Beginning tablet copy session from remote peer at address 127.17.52.190:39093
I20260927 03:18:14.660396 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.652596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:42786)
0927 03:18:14.652744 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.652755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.652830 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.652971 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.652972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.653167 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.653221 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.653223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.654440 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.654441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.655681 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.655685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.656399 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.656399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.656484 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.657057 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.657063 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.657484 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.659283 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.659284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.659285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.659500 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.659716 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.659717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.659717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.659780 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.660046 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.660047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.660093 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.660162 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.660303 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260927 03:18:14.660400 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.652676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.17.52.190:39093)
0927 03:18:14.652729 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.652732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.652812 (+ 80us) server_negotiation.cc:407] Connection header received
0927 03:18:14.652898 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.652899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.652910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.652955 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.653302 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.654325 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.655821 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.656196 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.656269 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.657583 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.657585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.657586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.657594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.659198 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.659581 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.659582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.659583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.659644 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.659849 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.659850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.659851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.659988 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.660058 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.660225 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.660312 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:14.661137 23755 tablet_copy_service.cc:140] P e6e2eac3378c45d7a126d38846438bbc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 01a07a3a10bf43359a40cf357b1f4718 ({username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:42786)
I20260927 03:18:14.661204 23755 tablet_copy_service.cc:161] P e6e2eac3378c45d7a126d38846438bbc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 01a07a3a10bf43359a40cf357b1f4718 at {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:42786: session id = 01a07a3a10bf43359a40cf357b1f4718-00000000000000000000000000000000
I20260927 03:18:14.661839 23755 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:18:14.662343 24390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:18:14.663612 24390 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: tablet copy: Starting download of 0 data blocks...
I20260927 03:18:14.663759 24390 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: tablet copy: Starting download of 1 WAL segments...
I20260927 03:18:14.664425 24390 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:18:14.665339 23755 tablet_copy_service.cc:342] P e6e2eac3378c45d7a126d38846438bbc: Request end of tablet copy session 01a07a3a10bf43359a40cf357b1f4718-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:42786
I20260927 03:18:14.665395 23755 tablet_copy_service.cc:433] P e6e2eac3378c45d7a126d38846438bbc: ending tablet copy session 01a07a3a10bf43359a40cf357b1f4718-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 01a07a3a10bf43359a40cf357b1f4718
I20260927 03:18:14.667647 24390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.668720 24502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.668728 24505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.669184 24390 server_base.cc:1034] running on GCE node
W20260927 03:18:14.672991 24503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.673206 24390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.673367 24390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.674515 24390 hybrid_clock.cc:648] HybridClock initialized: now 1790479094674502 us; error 29 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.676038 24390 init.cc:377] Logged in from keytab as oryx/127.17.52.188@KRBTEST.COM (short username oryx)
I20260927 03:18:14.676304 24390 webserver.cc:460] Webserver started at http://127.17.52.188:39817/ using document root <none> and password file <none>
I20260927 03:18:14.676422 24390 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.676462 24390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.677225 24390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:18:14.677817 24512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.677922 24390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:14.677954 24390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "01a07a3a10bf43359a40cf357b1f4718"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "b79c07f03e76897cd7c5d0bba8201e66"
server_key_iv: "eb0e5d8e4f857058759d4d663691a60d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.678009 24390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.701918 24390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.703734 24390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.706868 24390 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.188:46271
I20260927 03:18:14.706911 24564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.188:46271 every 8 connection(s)
I20260927 03:18:14.707195 24390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
I20260927 03:18:14.709376 24565 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:18:14.709892 24565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Bootstrap starting.
I20260927 03:18:14.712141 24565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:18:14.712352 24565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Bootstrap complete.
I20260927 03:18:14.712533 24565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
I20260927 03:18:14.712637 24565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01a07a3a10bf43359a40cf357b1f4718, State: Initialized, Role: LEARNER
I20260927 03:18:14.712678 24565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } }
I20260927 03:18:14.712836 24565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:14.712841 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:14.713090 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:18:14.714226 24579 catalog_manager.cc:1259] Loaded cluster ID: 5a1b1a494d1745b9a0a4b457d099901d
I20260927 03:18:14.714314 24579 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: loading cluster ID for follower catalog manager: success
I20260927 03:18:14.715559 24579 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: acquiring CA information for follower catalog manager: success
I20260927 03:18:14.715894 24579 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 03:18:14.738221 24484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:14.738409 24484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:14.738430 24484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:14.739475 24484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:14.739507 24484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:14.739532 24484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:14.740963 24484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:41095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=43181
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:14.741142 24484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:14.741314 24484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.745900 24585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.745930 24584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.746048 24484 server_base.cc:1034] running on GCE node
W20260927 03:18:14.745930 24587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.746344 24484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.746526 24484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.747725 24484 hybrid_clock.cc:648] HybridClock initialized: now 1790479094747721 us; error 34 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.750108 24484 init.cc:377] Logged in from keytab as oryx/127.17.52.129@KRBTEST.COM (short username oryx)
I20260927 03:18:14.750504 24484 webserver.cc:460] Webserver started at http://127.17.52.129:43181/ using document root <none> and password file <none>
I20260927 03:18:14.750732 24484 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.750813 24484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.752028 24484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:14.752543 24594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.752708 24484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.752801 24484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "b6e1ad773ea544168a227898f0a88bb5"
format_stamp: "Formatted at 2026-09-27 03:18:12 on dist-test-slave-7j43"
server_key: "89c97687a874242028c01349fa481c4a"
server_key_iv: "7fb1cb5d155b3d84b7f498711359fb2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.753109 24484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.767593 24484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.769549 24484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:14.769709 24484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.769918 24484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:14.770237 24484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:14.770287 24484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.770323 24484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:14.770351 24484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.776052 24484 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:41095
I20260927 03:18:14.776130 24707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:41095 every 8 connection(s)
I20260927 03:18:14.776445 24484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:14.780465 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24484
I20260927 03:18:14.780640 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23931
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
I20260927 03:18:14.788065 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:34295
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:39093
--webserver_port=35019
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:18:14.790287 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50729 (local address 127.17.52.190:39093)
0927 03:18:14.777613 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.777616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.777652 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:18:14.778605 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.778607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.778621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.778712 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.780157 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.780725 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782172 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.782462 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782500 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.786519 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.786520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.786520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.786528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.788459 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789230 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.789231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.789232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.789288 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789520 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.789521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.789521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.789675 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.789724 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.790110 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.790187 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:18:14.790318 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:40921 (local address 127.17.52.189:40151)
0927 03:18:14.777220 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.777221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.777490 (+ 269us) server_negotiation.cc:407] Connection header received
0927 03:18:14.778618 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.778619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.778631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.778660 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.779562 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.780057 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782019 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.782305 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782343 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.786518 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.786520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.786520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.786528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.788210 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.788843 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.788845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.788845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.788915 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789247 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.789249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.789249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.789827 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.789888 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.790139 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.790224 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:14.790750 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:50729)
0927 03:18:14.777589 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.777598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.778485 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.778758 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.778761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.779271 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.779458 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.779464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.781316 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.781317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.782095 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.782096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.782569 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.782621 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.784079 (+ 1458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.784085 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.786409 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.788886 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.788887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.788888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.789144 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.789384 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.789385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.789448 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.789813 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.789819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.789967 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.790088 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.790241 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260927 03:18:14.790759 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.17.52.129:40921)
0927 03:18:14.777344 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.777393 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.778484 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.778765 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.778766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.779275 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.779458 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.779464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.780176 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.780176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.781922 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.781924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.782421 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.782422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.782559 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.784047 (+ 1488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.784623 (+ 576us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.786409 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.788314 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.788321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.788324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.788720 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.788999 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.789000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.789155 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.789923 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.789924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.789969 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.790438 (+ 469us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.790541 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":6,"thread_start_us":87,"threads_started":1}
I20260927 03:18:14.791009 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:33253 (local address 127.17.52.188:46271)
0927 03:18:14.777989 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.777990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.778000 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:14.778668 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.778670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.778683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.778746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.782372 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.783469 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.784435 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.784839 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.784910 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.787279 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.787280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.787281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.787289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.789429 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789870 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.789871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.789872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.789935 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.790167 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.790168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.790168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.790303 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.790358 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.790819 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.790910 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260927 03:18:14.790761 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.777648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.17.52.129:33253)
0927 03:18:14.777763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.777771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.778498 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.778795 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.778796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.779234 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.779458 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.779463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.783557 (+ 4094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.783558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.784333 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.784335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.784967 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.784968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.785044 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.785520 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.785525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.787168 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.789512 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.789513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.789513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.789782 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.790009 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.790010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.790010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.790091 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.790398 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.790399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.790442 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.790510 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.790587 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260927 03:18:14.791246 24709 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:14.791337 24709 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.791456 24710 heartbeater.cc:347] Connected to a master server at 127.17.52.189:40151
I20260927 03:18:14.791497 24708 heartbeater.cc:347] Connected to a master server at 127.17.52.188:46271
I20260927 03:18:14.791497 24710 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.791536 24709 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
I20260927 03:18:14.791613 24710 heartbeater.cc:511] Master 127.17.52.189:40151 requested a full tablet report, sending...
I20260927 03:18:14.791546 24708 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.791759 24708 heartbeater.cc:511] Master 127.17.52.188:46271 requested a full tablet report, sending...
I20260927 03:18:14.792074 23735 ts_manager.cc:194] Re-registered known tserver with Master: b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
I20260927 03:18:14.792196 24529 ts_manager.cc:194] Registered new tserver with Master: b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
I20260927 03:18:14.792522 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:50729
I20260927 03:18:14.792536 24336 ts_manager.cc:194] Registered new tserver with Master: b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
W20260927 03:18:14.871228 24716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:14.871466 24716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:14.871507 24716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:14.873106 24716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:14.873165 24716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:14.873210 24716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:14.874871 24716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:34295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=35019
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:14.875149 24716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:14.875365 24716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:14.880259 24721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.880252 24722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:14.880252 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:14.880514 24716 server_base.cc:1034] running on GCE node
I20260927 03:18:14.880826 24716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:14.881063 24716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:14.882234 24716 hybrid_clock.cc:648] HybridClock initialized: now 1790479094882221 us; error 29 us; skew 500 ppm
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:14.884693 24716 init.cc:377] Logged in from keytab as oryx/127.17.52.130@KRBTEST.COM (short username oryx)
I20260927 03:18:14.885159 24716 webserver.cc:460] Webserver started at http://127.17.52.130:35019/ using document root <none> and password file <none>
I20260927 03:18:14.885396 24716 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:14.885483 24716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:14.886847 24716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:14.887444 24731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.887593 24716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:14.887676 24716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
format_stamp: "Formatted at 2026-09-27 03:18:13 on dist-test-slave-7j43"
server_key: "fb8b3d2c65acaab30f344e1b01134a0d"
server_key_iv: "ce41ee24b24b7f295630543bfa2303fd"
server_key_version: "encryptionkey@0"
I20260927 03:18:14.887969 24716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:14.922011 24716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:14.923779 24716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:14.923884 24716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:14.924063 24716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:14.924371 24716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:14.924405 24716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.924427 24716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:14.924475 24716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:14.930533 24716 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:34295
I20260927 03:18:14.930586 24844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:34295 every 8 connection(s)
I20260927 03:18:14.930949 24716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:14.935702 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24716
I20260927 03:18:14.935849 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24066
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
I20260927 03:18:14.942468 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.931917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:43617 (local address 127.17.52.189:40151)
0927 03:18:14.931960 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.931962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.932186 (+ 224us) server_negotiation.cc:407] Connection header received
0927 03:18:14.933029 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.933030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.933042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.933075 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.933774 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934536 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935645 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.935947 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935987 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.939558 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.939559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.939559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.939567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.941218 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941608 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.941609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.941609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.941656 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941852 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.941852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.941853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.941980 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.942018 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.942296 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.942374 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:18:14.942480 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.932064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36445 (local address 127.17.52.190:39093)
0927 03:18:14.932109 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.932111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.932195 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:18:14.933028 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.933030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.933042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.933075 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.933781 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934542 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935645 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.935947 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935987 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.938966 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.938967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.938968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.938976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.940931 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941410 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.941411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.941411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.941463 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941735 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.941735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.941736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.941886 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.941930 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.942292 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.942377 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 03:18:14 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479094, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
I20260927 03:18:14.943187 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.932007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:36445)
0927 03:18:14.932139 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.932148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.932928 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.933160 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.933165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.933530 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.933667 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.933673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934632 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.934633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.935546 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.935548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.936111 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.936112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.936213 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.936939 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.936943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.938858 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.941024 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.941032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.941325 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.941536 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.941537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.941657 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.941974 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.941980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.942135 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.942779 (+ 644us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.942944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":47,"thread_start_us":62,"threads_started":1}
I20260927 03:18:14.943200 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:42637
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:39093
--webserver_port=38557
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:18:14.943621 24846 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:14.943703 24846 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.943882 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.931839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.17.52.130:43617)
0927 03:18:14.932122 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.932134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.932931 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.933160 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.933165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.933527 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.933667 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.933673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934632 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.934633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.935548 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.935549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.936070 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.936071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.936213 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.936859 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.936873 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.939484 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.941312 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.941313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.941530 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.941730 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.941731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.941731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.941782 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.942060 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.942060 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.942135 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.942919 (+ 784us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.942991 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":31,"thread_start_us":68,"threads_started":1}
I20260927 03:18:14.944617 24847 heartbeater.cc:347] Connected to a master server at 127.17.52.189:40151
I20260927 03:18:14.944672 24847 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.944742 24847 heartbeater.cc:511] Master 127.17.52.189:40151 requested a full tablet report, sending...
I20260927 03:18:14.943912 24846 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
I20260927 03:18:14.945159 23735 ts_manager.cc:194] Re-registered known tserver with Master: b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
I20260927 03:18:14.945220 24336 ts_manager.cc:194] Registered new tserver with Master: b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
I20260927 03:18:14.945650 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:36445
I20260927 03:18:14.946151 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.931618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.17.52.130:59887)
0927 03:18:14.931751 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:14.931793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:14.932911 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:14.933160 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:14.933163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:14.933478 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:14.933666 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.933669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934500 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.934500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.935425 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:14.935427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.941127 (+ 5700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.941128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:14.941211 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.941653 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:14.941658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:14.943181 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:14.944996 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.944996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.944997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.945217 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.945448 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:14.945449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:14.945449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:14.945504 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:14.945816 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:14.945817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:14.945858 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:18:14.945944 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:18:14.946048 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260927 03:18:14.946152 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:14.931613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:59887 (local address 127.17.52.188:46271)
0927 03:18:14.931654 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:14.931656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:14.931915 (+ 259us) server_negotiation.cc:407] Connection header received
0927 03:18:14.933025 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:14.933026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:14.933043 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:14.933093 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:14.933759 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.934404 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935504 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:14.935855 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:14.935922 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:14.943303 (+ 7381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:14.943304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:14.943304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:14.943313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:14.944925 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.945308 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.945310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.945310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.945371 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:14.945558 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:14.945559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:14.945560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:14.945702 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:14.945785 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:14.945982 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 03:18:14.946064 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:18:14.946520 24845 heartbeater.cc:347] Connected to a master server at 127.17.52.188:46271
I20260927 03:18:14.946574 24845 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:14.946645 24845 heartbeater.cc:511] Master 127.17.52.188:46271 requested a full tablet report, sending...
I20260927 03:18:14.946955 24529 ts_manager.cc:194] Registered new tserver with Master: b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
W20260927 03:18:15.026070 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:15.026286 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:15.026311 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:15.027542 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:15.027596 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:15.027626 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:15.029103 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=38557
--tserver_master_addrs=127.17.52.190:39093,127.17.52.189:40151,127.17.52.188:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:15.029289 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:15.029500 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:15.034379 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:15.034392 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:15.034392 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:15.034461 24852 server_base.cc:1034] running on GCE node
I20260927 03:18:15.034776 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:15.035015 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:15.036185 24852 hybrid_clock.cc:648] HybridClock initialized: now 1790479095036171 us; error 29 us; skew 500 ppm
Sep 27 03:18:15 dist-test-slave-7j43 krb5kdc[23690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479095, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:15.038563 24852 init.cc:377] Logged in from keytab as oryx/127.17.52.131@KRBTEST.COM (short username oryx)
I20260927 03:18:15.039013 24852 webserver.cc:460] Webserver started at http://127.17.52.131:38557/ using document root <none> and password file <none>
I20260927 03:18:15.039229 24852 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:15.039310 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:15.040632 24852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:15.041183 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:15.041355 24852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:15.041450 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "f7cd6500ef184de096d1f8843bc3e5e1"
format_stamp: "Formatted at 2026-09-27 03:18:14 on dist-test-slave-7j43"
server_key: "af326f89e826ba44eb860fb1233eca54"
server_key_iv: "a0731b2ba316545bcd029afcd9e69297"
server_key_version: "encryptionkey@0"
I20260927 03:18:15.041780 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:15.049928 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:15.051852 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:15.051990 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:15.052197 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:15.052521 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:15.052589 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:15.052621 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:15.052647 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:15.058188 24852 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42637
I20260927 03:18:15.058256 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42637 every 8 connection(s)
I20260927 03:18:15.058600 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:15.058995 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 24852
Sep 27 03:18:15 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479095, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
Sep 27 03:18:15 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479095, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.189@KRBTEST.COM
Sep 27 03:18:15 dist-test-slave-7j43 krb5kdc[23690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479095, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.188@KRBTEST.COM
I20260927 03:18:15.069080 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.060726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.17.52.188:46271)
0927 03:18:15.061032 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.061034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.061260 (+ 226us) server_negotiation.cc:407] Connection header received
0927 03:18:15.061351 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.061352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.061399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.061442 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.062127 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.062885 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.063966 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.064259 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.064302 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.065915 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.065916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.065917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.065924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.067705 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.068076 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.068077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.068077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.068134 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.068344 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.068345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.068346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.068471 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.068546 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.068841 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.068937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":66,"threads_started":1}
I20260927 03:18:15.072377 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.059656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:55049 (local address 127.17.52.188:46271)
0927 03:18:15.060059 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.060061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.060073 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:18:15.060716 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.060717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.060727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.060783 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.061799 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.062588 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065201 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.065496 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065557 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.069047 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.069048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.069049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.069056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.071149 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071526 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.071527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.071528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.071580 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071773 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.071774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.071774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.071880 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.071959 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.072204 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.072285 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260927 03:18:15.072515 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.060157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:59057 (local address 127.17.52.189:40151)
0927 03:18:15.060197 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.060199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.060206 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:15.060458 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.060459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.060469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.060544 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.062673 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.063355 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065037 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.065328 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065382 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.067950 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.067951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.067952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.067959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.070793 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071282 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.071283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.071283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.071333 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071655 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.071657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.071657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.071803 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.071865 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.072323 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.072398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:15.072811 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.059589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.17.52.131:55049)
0927 03:18:15.059866 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.059875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.060285 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.061120 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.061121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.061391 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:15.061694 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.061701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.064326 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.064327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.065122 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.065124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.065832 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.065890 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.066613 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:15.066619 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:15.068951 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:15.071231 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.071232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.071452 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.071646 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.071647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.071705 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.071963 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:15.071964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:15.072067 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.072178 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.072349 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":56,"threads_started":1}
I20260927 03:18:15.072834 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.059740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.17.52.131:59057)
0927 03:18:15.059956 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.059965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.060296 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.060838 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.060839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.061774 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:15.061841 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.061842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.063447 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.063448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.064243 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.064245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.065607 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.065608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.065675 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.066053 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:15.066058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:15.067849 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:15.070887 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.070895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.070897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.071205 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.071402 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.071403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.071567 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.071894 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:15.071901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:15.072075 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.072171 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.072702 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":357,"thread_start_us":61,"threads_started":1}
I20260927 03:18:15.073582 24983 heartbeater.cc:347] Connected to a master server at 127.17.52.189:40151
I20260927 03:18:15.073599 24981 heartbeater.cc:347] Connected to a master server at 127.17.52.188:46271
I20260927 03:18:15.073711 24983 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:15.073711 24981 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:15.073918 24983 heartbeater.cc:511] Master 127.17.52.189:40151 requested a full tablet report, sending...
I20260927 03:18:15.073918 24981 heartbeater.cc:511] Master 127.17.52.188:46271 requested a full tablet report, sending...
I20260927 03:18:15.074357 24529 ts_manager.cc:194] Registered new tserver with Master: f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637)
I20260927 03:18:15.074393 24336 ts_manager.cc:194] Registered new tserver with Master: f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637)
I20260927 03:18:15.074851 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.059309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:33611 (local address 127.17.52.190:39093)
0927 03:18:15.059382 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.059384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.059624 (+ 240us) server_negotiation.cc:407] Connection header received
0927 03:18:15.060404 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.060405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.060419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.060451 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.062074 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.062786 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.064232 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.064516 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.064557 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.068376 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.068377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.068377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.068384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.071288 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.073174 (+ 1886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.073175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.073175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.073233 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.073783 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.073784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.073784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.073930 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.073988 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.074636 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.074715 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:18:15.075248 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.059328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:33611)
0927 03:18:15.059485 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.059532 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.060280 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.060612 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.060617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.061947 (+ 1330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:15.061990 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.061991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.062901 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.062902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.064135 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.064137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.064670 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.064671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.064818 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.065489 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:15.065499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:15.068276 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:15.071386 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.071387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.071388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.073089 (+ 1701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.073568 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:15.073569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:15.073569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:15.073666 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:15.074024 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:15.074024 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:15.074066 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.074786 (+ 720us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.074954 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260927 03:18:15.075605 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.060609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.17.52.189:40151)
0927 03:18:15.061264 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.061266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.061274 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:15.061294 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.061295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.061306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.061610 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.062141 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.062879 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.066975 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.067302 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.068456 (+ 1154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.069675 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.069677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.069677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.069683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.074068 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.074531 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.074532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.074533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.074593 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.074823 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.074823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.074824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.074951 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.075393 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.075431 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.075513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":59,"threads_started":1}
I20260927 03:18:15.075683 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.060155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.17.52.190:39093)
0927 03:18:15.060455 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.060457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.060662 (+ 205us) server_negotiation.cc:407] Connection header received
0927 03:18:15.060793 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.060794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.060807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.060852 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.063147 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.063883 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.067699 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.068021 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.068063 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.071519 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.071520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.071520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.071527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.073429 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.074763 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.074764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.074764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.074856 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.075075 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.075076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.075076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.075209 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.075299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.075489 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.075561 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260927 03:18:15.075755 24982 heartbeater.cc:347] Connected to a master server at 127.17.52.190:39093
I20260927 03:18:15.075790 24982 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:15.075850 24982 heartbeater.cc:511] Master 127.17.52.190:39093 requested a full tablet report, sending...
I20260927 03:18:15.076056 23735 ts_manager.cc:194] Re-registered known tserver with Master: f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637)
I20260927 03:18:15.076601 23735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:33611
I20260927 03:18:15.077961 23735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:15.079522 23735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:15.087076 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:34295 (local address 127.0.0.1:42586)
0927 03:18:15.083727 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.083737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.083784 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.084025 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.084026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.084027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.084061 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.084062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.085282 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.085283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.086184 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.086186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.086837 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.086838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.086897 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.086902 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.086958 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.086997 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":96,"threads_started":1}
I20260927 03:18:15.087080 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.17.52.130:34295)
0927 03:18:15.083639 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.083643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.083837 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:18:15.083883 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.083886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.083901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.083965 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.084130 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.085202 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.086271 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.086743 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.086826 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.086901 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.086972 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.087006 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260927 03:18:15.087229 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:41095 (local address 127.0.0.1:52480)
0927 03:18:15.083345 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.083357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.083398 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.083745 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.083746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.083748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.083790 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.083791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.084663 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.084664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.085554 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.085556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.087006 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.087007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.087074 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.087081 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.087112 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.087157 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":9}
I20260927 03:18:15.087369 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42637 (local address 127.0.0.1:38438)
0927 03:18:15.083381 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.083392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.083453 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.083652 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.083653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.083655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.083709 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.083711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.085182 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.085183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.086129 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.086131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.087006 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.087007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.087074 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.087081 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.087112 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.087156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260927 03:18:15.087734 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.17.52.129:41095)
0927 03:18:15.083570 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.083575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.083586 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:15.083628 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.083630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.083646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.083865 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.083910 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.084538 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.086236 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.086904 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.087523 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.087600 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.087623 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.087661 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
I20260927 03:18:15.087977 24779 tablet_service.cc:1467] Processing CreateTablet for tablet 452794716ebc4d7bb031dc3e5685603f (DEFAULT_TABLE table=test-table [id=ddb3a18c4eeb4eec9cf2546ea1225a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:15.088366 24779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452794716ebc4d7bb031dc3e5685603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:15.089007 24642 tablet_service.cc:1467] Processing CreateTablet for tablet 452794716ebc4d7bb031dc3e5685603f (DEFAULT_TABLE table=test-table [id=ddb3a18c4eeb4eec9cf2546ea1225a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:15.089398 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452794716ebc4d7bb031dc3e5685603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:15.092003 25007 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Bootstrap starting.
I20260927 03:18:15.092649 25007 tablet_bootstrap.cc:654] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:15.092798 25008 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Bootstrap starting.
I20260927 03:18:15.093053 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.083219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.17.52.131:42637)
0927 03:18:15.083373 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.083378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.083441 (+ 63us) server_negotiation.cc:407] Connection header received
0927 03:18:15.083531 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.083534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.083553 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.083799 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.083855 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.085101 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.086209 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.086905 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.087404 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.087487 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.092912 (+ 5425us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.092963 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260927 03:18:15.093617 25008 tablet_bootstrap.cc:654] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:15.093628 24915 tablet_service.cc:1467] Processing CreateTablet for tablet 452794716ebc4d7bb031dc3e5685603f (DEFAULT_TABLE table=test-table [id=ddb3a18c4eeb4eec9cf2546ea1225a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:15.093871 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452794716ebc4d7bb031dc3e5685603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:15.094063 25008 log.cc:824] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:15.094367 25007 log.cc:824] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:15.095072 25007 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: No bootstrap required, opened a new log
I20260927 03:18:15.095161 25007 ts_tablet_manager.cc:1397] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 03:18:15.095641 25008 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: No bootstrap required, opened a new log
I20260927 03:18:15.095696 25008 ts_tablet_manager.cc:1397] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:18:15.096932 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.092752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.0.0.1:35192)
0927 03:18:15.092793 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.092803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.092840 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.093069 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.093070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.093071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.093104 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.093106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.093973 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.093974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.095030 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.095033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.096668 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.096669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.096746 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.096750 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.096808 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.096852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:18:15.097268 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.092866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.17.52.188:46271)
0927 03:18:15.092915 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.092917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.092928 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:15.092954 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.092955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.092977 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.093051 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.093176 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.093882 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.095126 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.095716 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.095835 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.095908 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.097150 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.097195 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:18:15.097534 24539 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 03:18:15.097548 25007 raft_consensus.cc:348] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.097698 25007 raft_consensus.cc:374] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:15.097744 25007 raft_consensus.cc:729] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d16df85ae84ce6a39ef365f1f183f8, State: Initialized, Role: FOLLOWER
I20260927 03:18:15.097874 25007 consensus_queue.cc:260] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.098142 25007 ts_tablet_manager.cc:1428] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:18:15.098246 25008 raft_consensus.cc:348] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.098414 25008 raft_consensus.cc:374] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:15.098456 25008 raft_consensus.cc:729] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e1ad773ea544168a227898f0a88bb5, State: Initialized, Role: FOLLOWER
I20260927 03:18:15.098569 25008 consensus_queue.cc:260] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.098680 24846 heartbeater.cc:503] Master 127.17.52.190:39093 was elected leader, sending a full tablet report...
I20260927 03:18:15.099114 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e2eac3378c45d7a126d38846438bbc. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:15.099223 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:18:15.099367 25008 ts_tablet_manager.cc:1428] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:18:15.099514 24709 heartbeater.cc:503] Master 127.17.52.190:39093 was elected leader, sending a full tablet report...
I20260927 03:18:15.099620 25011 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Bootstrap starting.
I20260927 03:18:15.101300 25011 tablet_bootstrap.cc:654] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:15.101588 25011 log.cc:824] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:15.102242 25011 tablet_bootstrap.cc:492] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: No bootstrap required, opened a new log
I20260927 03:18:15.102293 25011 ts_tablet_manager.cc:1397] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:18:15.104255 25011 raft_consensus.cc:348] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.104372 25011 raft_consensus.cc:374] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:15.104391 25011 raft_consensus.cc:729] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cd6500ef184de096d1f8843bc3e5e1, State: Initialized, Role: FOLLOWER
I20260927 03:18:15.104538 25011 consensus_queue.cc:260] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.104837 25011 ts_tablet_manager.cc:1428] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:18:15.104959 24982 heartbeater.cc:503] Master 127.17.52.190:39093 was elected leader, sending a full tablet report...
W20260927 03:18:15.181666 24848 tablet.cc:2369] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:18:15.237108 25018 raft_consensus.cc:482] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:15.237255 25018 raft_consensus.cc:504] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.237601 25018 leader_election.cc:288] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095), b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
I20260927 03:18:15.241511 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.238008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:37717 (local address 127.17.52.130:34295)
0927 03:18:15.238044 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.238046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.238097 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:18:15.238124 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.238124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.238134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.238181 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.238388 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.239144 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.240603 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.241153 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.241228 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.241236 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.241401 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.241442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:15.241511 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.237888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:34953 (local address 127.17.52.129:41095)
0927 03:18:15.237951 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.237953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.237983 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:18:15.238127 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.238128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.238139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.238189 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:15.238421 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.239144 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.240619 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.241154 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.241228 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.241236 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.241401 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.241439 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:18:15.241529 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.237824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:41095 (local address 127.17.52.131:34953)
0927 03:18:15.237895 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.237921 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.237974 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.238203 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.238204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.238206 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.238310 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.238337 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.239215 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.239216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.240508 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.240510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.241233 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.241234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.241323 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.241327 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.241385 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.241435 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":49}
I20260927 03:18:15.241544 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.237946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:34295 (local address 127.17.52.131:37717)
0927 03:18:15.237990 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:15.238000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:15.238038 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:15.238231 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:15.238232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:15.238234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:15.238304 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.238306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.239226 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.239227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.240508 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:15.240510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.241233 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.241234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:15.241323 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.241328 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:18:15.241385 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:18:15.241435 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":93}
I20260927 03:18:15.241938 24662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452794716ebc4d7bb031dc3e5685603f" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e1ad773ea544168a227898f0a88bb5" is_pre_election: true
I20260927 03:18:15.242094 24662 raft_consensus.cc:2454] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 0.
I20260927 03:18:15.242115 24799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452794716ebc4d7bb031dc3e5685603f" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" is_pre_election: true
I20260927 03:18:15.242282 24799 raft_consensus.cc:2454] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 0.
I20260927 03:18:15.242312 24869 leader_election.cc:302] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e1ad773ea544168a227898f0a88bb5, f7cd6500ef184de096d1f8843bc3e5e1; no voters:
I20260927 03:18:15.242446 25018 raft_consensus.cc:2781] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:15.242523 25018 raft_consensus.cc:482] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:15.242559 25018 raft_consensus.cc:3037] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:15.243251 25018 raft_consensus.cc:504] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.243399 25018 leader_election.cc:288] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 election: Requested vote from peers b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095), b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295)
I20260927 03:18:15.243582 24662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452794716ebc4d7bb031dc3e5685603f" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e1ad773ea544168a227898f0a88bb5"
I20260927 03:18:15.243594 24799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452794716ebc4d7bb031dc3e5685603f" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
I20260927 03:18:15.243687 24662 raft_consensus.cc:3037] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:15.243685 24799 raft_consensus.cc:3037] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:15.244418 24799 raft_consensus.cc:2454] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 1.
I20260927 03:18:15.244524 24662 raft_consensus.cc:2454] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 1.
I20260927 03:18:15.244611 24869 leader_election.cc:302] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7d16df85ae84ce6a39ef365f1f183f8, f7cd6500ef184de096d1f8843bc3e5e1; no voters:
I20260927 03:18:15.244696 25018 raft_consensus.cc:2781] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:15.244837 25018 raft_consensus.cc:686] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 LEADER]: Becoming Leader. State: Replica: f7cd6500ef184de096d1f8843bc3e5e1, State: Running, Role: LEADER
I20260927 03:18:15.244932 25018 consensus_queue.cc:237] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } }
I20260927 03:18:15.245826 23735 catalog_manager.cc:5495] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:15.246469 24493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: NON_VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: true } } }
I20260927 03:18:15.246570 24493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:15.247107 24493 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: attempting to promote NON_VOTER 01a07a3a10bf43359a40cf357b1f4718 to VOTER
I20260927 03:18:15.247241 24493 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } }
I20260927 03:18:15.247845 24346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:18:15.247915 24539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 LEARNER]: Refusing update from remote peer e6e2eac3378c45d7a126d38846438bbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:18:15.248086 24298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 03:18:15.248191 25021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 03:18:15.248775 24493 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.248900 24539 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.249529 24493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.249605 24493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:15.249681 24493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.249742 24493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:15.249805 24493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.249852 24493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:15.249934 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.250007 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:18:15.250078 24346 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.250732 25023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e2eac3378c45d7a126d38846438bbc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:15.250823 25023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:18:15.266256 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.264031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.17.52.131:42637)
0927 03:18:15.264089 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.264091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.264099 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:15.264140 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.264141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.264157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.264218 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:15.264338 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.264905 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.265599 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.265786 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.265833 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.265954 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:15.266049 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:15.266107 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.266137 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.266152 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:18:15.267982 24799 raft_consensus.cc:1261] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Refusing update from remote peer f7cd6500ef184de096d1f8843bc3e5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:15.267982 24662 raft_consensus.cc:1261] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Refusing update from remote peer f7cd6500ef184de096d1f8843bc3e5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:15.268373 25018 consensus_queue.cc:1035] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:15.268522 25019 consensus_queue.cc:1035] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:15.271018 25025 mvcc.cc:205] Tried to move back new op lower bound from 7333802374214901760 to 7333802374123851776. Current Snapshot: MvccSnapshot[applied={T|T < 7333802374214901760}]
I20260927 03:18:15.271229 25026 mvcc.cc:205] Tried to move back new op lower bound from 7333802374214901760 to 7333802374123851776. Current Snapshot: MvccSnapshot[applied={T|T < 7333802374214901760}]
I20260927 03:18:15.271384 25027 mvcc.cc:205] Tried to move back new op lower bound from 7333802374214901760 to 7333802374123851776. Current Snapshot: MvccSnapshot[applied={T|T < 7333802374214901760}]
I20260927 03:18:15.274674 23734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:15.274765 23734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:15.276340 23734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6e2eac3378c45d7a126d38846438bbc: Sending DeleteTablet for 3 replicas of tablet 452794716ebc4d7bb031dc3e5685603f
I20260927 03:18:15.276654 24642 tablet_service.cc:1514] Processing DeleteTablet for tablet 452794716ebc4d7bb031dc3e5685603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:15 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:52480
I20260927 03:18:15.276674 24779 tablet_service.cc:1514] Processing DeleteTablet for tablet 452794716ebc4d7bb031dc3e5685603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:15 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:42586
I20260927 03:18:15.277315 24915 tablet_service.cc:1514] Processing DeleteTablet for tablet 452794716ebc4d7bb031dc3e5685603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:15 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:38438
I20260927 03:18:15.277527 25038 tablet_replica.cc:330] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: stopping tablet replica
I20260927 03:18:15.277622 25038 raft_consensus.cc:2229] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:15.277815 25035 tablet_replica.cc:330] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: stopping tablet replica
I20260927 03:18:15.277840 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 23703
I20260927 03:18:15.278216 25035 raft_consensus.cc:2229] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:15.278330 25035 raft_consensus.cc:2258] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:15.278808 25035 ts_tablet_manager.cc:1905] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:15.280737 25035 ts_tablet_manager.cc:1918] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:15.280826 25035 log.cc:1196] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/452794716ebc4d7bb031dc3e5685603f
I20260927 03:18:15.281091 25035 ts_tablet_manager.cc:1939] T 452794716ebc4d7bb031dc3e5685603f P b6e1ad773ea544168a227898f0a88bb5: Deleting consensus metadata
I20260927 03:18:15.281512 25036 tablet_replica.cc:330] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: stopping tablet replica
I20260927 03:18:15.281626 25036 raft_consensus.cc:2229] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:15.281731 25036 raft_consensus.cc:2258] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:15.282224 25036 ts_tablet_manager.cc:1905] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:15.284121 25036 ts_tablet_manager.cc:1918] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:15.284194 25036 log.cc:1196] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/452794716ebc4d7bb031dc3e5685603f
I20260927 03:18:15.284442 25036 ts_tablet_manager.cc:1939] T 452794716ebc4d7bb031dc3e5685603f P b7d16df85ae84ce6a39ef365f1f183f8: Deleting consensus metadata
I20260927 03:18:15.285169 25038 raft_consensus.cc:2258] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:18:15.285729 24733 connection.cc:535] server connection from 127.0.0.1:42586 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:18:15.285837 25038 ts_tablet_manager.cc:1905] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:18:15.286137 24597 connection.cc:535] server connection from 127.0.0.1:52480 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:18:15.287467 25038 ts_tablet_manager.cc:1918] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:15.287531 25038 log.cc:1196] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/452794716ebc4d7bb031dc3e5685603f
I20260927 03:18:15.287778 25038 ts_tablet_manager.cc:1939] T 452794716ebc4d7bb031dc3e5685603f P f7cd6500ef184de096d1f8843bc3e5e1: Deleting consensus metadata
W20260927 03:18:15.288291 24868 connection.cc:441] server connection from 127.0.0.1:38438 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38438 (request call id 1) could send its response
I20260927 03:18:15.290714 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.277022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.17.52.188:46271)
0927 03:18:15.277064 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.277066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.277191 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:18:15.277311 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.277312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.277324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.277383 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.278368 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.279120 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.280262 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.280606 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.280654 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.287044 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.287046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.287046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.287054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.289093 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.289567 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.289568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.289569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.289628 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.289879 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.289881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.289881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.290050 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.290145 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.290549 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.290619 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:15.296666 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.288708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.17.52.189:40151)
0927 03:18:15.288753 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.288755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.288771 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:18:15.288797 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.288798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.288828 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.288885 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.289610 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.290417 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.291642 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.291964 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.292051 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.293917 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.293919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.293919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.293927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.295446 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.295845 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.295846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.295847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.295904 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.296115 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.296117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.296117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.296241 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.296345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.296449 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.296526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:18:15.296916 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:15.288564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.17.52.188:46271)
0927 03:18:15.288670 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:15.288672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:15.288885 (+ 213us) server_negotiation.cc:407] Connection header received
0927 03:18:15.289537 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:15.289539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:15.289554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:15.289584 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:15.290331 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.291096 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.292558 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:15.292888 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:15.292959 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:15.294096 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:15.294097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:15.294098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:15.294104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:15.295695 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.296068 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.296069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.296069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.296117 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:15.296336 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:15.296337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:15.296337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:15.296452 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:15.296517 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:15.296746 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 03:18:15.296825 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 03:18:16.272647 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.272347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:47405)
0927 03:18:16.272482 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.272564 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260927 03:18:16.272845 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.272347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:60095)
0927 03:18:16.272480 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.272804 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260927 03:18:16.273003 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.272781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:46145)
0927 03:18:16.272893 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.272957 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":46,"threads_started":1}
W20260927 03:18:16.273144 24846 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:39093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
W20260927 03:18:16.273362 24982 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:39093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
W20260927 03:18:16.273402 24709 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:39093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.776726 25049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6e2eac3378c45d7a126d38846438bbc)
I20260927 03:18:16.776803 25049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } }
I20260927 03:18:16.776952 25049 leader_election.cc:288] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093), 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189:40151)
I20260927 03:18:16.777261 25051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6e2eac3378c45d7a126d38846438bbc)
I20260927 03:18:16.777357 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.777048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43268)
0927 03:18:16.777232 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.777305 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
W20260927 03:18:16.777491 24515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.777575 25051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } }
I20260927 03:18:16.777872 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.777781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43280)
0927 03:18:16.777817 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.777848 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:18:16.777953 24515 leader_election.cc:334] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093): Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.778072 25051 leader_election.cc:288] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093), 01a07a3a10bf43359a40cf357b1f4718 (127.17.52.188:46271)
I20260927 03:18:16.778136 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.777969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43288)
0927 03:18:16.778046 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.778101 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":44,"threads_started":1}
W20260927 03:18:16.778273 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.778764 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.778597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43296)
0927 03:18:16.778706 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.778733 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
W20260927 03:18:16.778841 24322 leader_election.cc:334] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093): Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.780509 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.777322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.17.52.189:40151)
0927 03:18:16.777475 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.777477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.777487 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:16.777789 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.777790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.777803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.777864 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:16.778002 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.778821 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.779794 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.780165 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.780245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.780256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.780397 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.780441 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260927 03:18:16.780544 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.777246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:40151 (local address 127.0.0.1:40062)
0927 03:18:16.777329 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.777341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:16.777680 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:16.777866 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:16.777867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:16.777869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:16.777924 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.777926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.778913 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.778914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.779690 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.779692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.780244 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.780245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.780315 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.780317 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:18:16.780389 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:18:16.780462 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":27,"thread_start_us":34,"threads_started":1}
I20260927 03:18:16.781039 24346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a07a3a10bf43359a40cf357b1f4718" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4abc28b0becc4465a9339e851140f6a2" is_pre_election: true
I20260927 03:18:16.781147 24346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a07a3a10bf43359a40cf357b1f4718 in term 1.
I20260927 03:18:16.781201 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.778248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.17.52.188:46271)
0927 03:18:16.778400 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.778402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.778459 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:18:16.778565 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.778566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.778575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.778640 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:16.778778 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.779386 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.780332 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.780882 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.780946 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.780955 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.781093 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.781133 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":112,"threads_started":1}
I20260927 03:18:16.781265 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.778183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:46271 (local address 127.0.0.1:35216)
0927 03:18:16.778382 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.778394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:16.778484 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:16.778655 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:16.778655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:16.778658 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:16.778699 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.778701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.779466 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.779467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.780222 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.780224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.780956 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.780957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.781023 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.781026 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:16.781073 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:18:16.781101 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 03:18:16.781299 24515 leader_election.cc:302] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01a07a3a10bf43359a40cf357b1f4718, 4abc28b0becc4465a9339e851140f6a2; no voters: e6e2eac3378c45d7a126d38846438bbc
I20260927 03:18:16.781369 25049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:18:16.781407 25049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e6e2eac3378c45d7a126d38846438bbc)
I20260927 03:18:16.781437 25049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:18:16.781490 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4abc28b0becc4465a9339e851140f6a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "01a07a3a10bf43359a40cf357b1f4718" is_pre_election: true
I20260927 03:18:16.782119 25049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } }
I20260927 03:18:16.782254 25049 leader_election.cc:288] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 election: Requested vote from peers e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093), 4abc28b0becc4465a9339e851140f6a2 (127.17.52.189:40151)
I20260927 03:18:16.782274 24539 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4abc28b0becc4465a9339e851140f6a2 in current term 2: Already voted for candidate 01a07a3a10bf43359a40cf357b1f4718 in this term.
I20260927 03:18:16.782450 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.782326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43312)
0927 03:18:16.782401 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.782427 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 03:18:16.782450 24346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a07a3a10bf43359a40cf357b1f4718" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4abc28b0becc4465a9339e851140f6a2"
I20260927 03:18:16.782464 24322 leader_election.cc:302] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4abc28b0becc4465a9339e851140f6a2; no voters: 01a07a3a10bf43359a40cf357b1f4718, e6e2eac3378c45d7a126d38846438bbc
I20260927 03:18:16.782524 24346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:18:16.782863 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.782778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43328)
0927 03:18:16.782825 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.782848 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:18:16.782953 24515 leader_election.cc:334] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093): Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
I20260927 03:18:16.783274 24346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a07a3a10bf43359a40cf357b1f4718 in term 2.
I20260927 03:18:16.783352 25051 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:18:16.783428 24515 leader_election.cc:302] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01a07a3a10bf43359a40cf357b1f4718, 4abc28b0becc4465a9339e851140f6a2; no voters: e6e2eac3378c45d7a126d38846438bbc
I20260927 03:18:16.783540 25049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:18:16.783682 25049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [term 2 LEADER]: Becoming Leader. State: Replica: 01a07a3a10bf43359a40cf357b1f4718, State: Running, Role: LEADER
I20260927 03:18:16.783761 25049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } }
I20260927 03:18:16.784078 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a07a3a10bf43359a40cf357b1f4718. Latest consensus state: current_term: 2 leader_uuid: "01a07a3a10bf43359a40cf357b1f4718" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:16.784178 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:16.784365 25061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:16.784793 25061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:16.784852 25061 catalog_manager.cc:1259] Loaded cluster ID: 5a1b1a494d1745b9a0a4b457d099901d
I20260927 03:18:16.784868 25061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:16.784976 25061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:16.785025 25061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Loaded TSK: 0
I20260927 03:18:16.785116 25061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:16.833397 24529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:16.834877 24529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:16.835789 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.835680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43348)
0927 03:18:16.835727 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.835756 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:18:16.835909 24346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [term 2 FOLLOWER]: Refusing update from remote peer 01a07a3a10bf43359a40cf357b1f4718: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 03:18:16.836189 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.836054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.0.0.1:43360)
0927 03:18:16.836107 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.836140 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:18:16.836135 25049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 03:18:16.836243 24515 consensus_peers.cc:487] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 -> Peer e6e2eac3378c45d7a126d38846438bbc (127.17.52.190:39093): Couldn't send request to peer e6e2eac3378c45d7a126d38846438bbc. Status: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:18:16.836735 25051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a07a3a10bf43359a40cf357b1f4718. Latest consensus state: current_term: 2 leader_uuid: "01a07a3a10bf43359a40cf357b1f4718" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:16.836835 25051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:18:16.837277 25049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "01a07a3a10bf43359a40cf357b1f4718" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:16.837374 25049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:16.837388 25051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "01a07a3a10bf43359a40cf357b1f4718" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e6e2eac3378c45d7a126d38846438bbc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 39093 } } peers { permanent_uuid: "4abc28b0becc4465a9339e851140f6a2" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 40151 } attrs { promote: false } } peers { permanent_uuid: "01a07a3a10bf43359a40cf357b1f4718" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 46271 } attrs { promote: false } } }
I20260927 03:18:16.837481 25051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4abc28b0becc4465a9339e851140f6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:18:16.842259 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42637 (local address 127.0.0.1:38454)
0927 03:18:16.839258 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.839267 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:16.839296 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:16.839592 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:16.839594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:16.839595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:16.839625 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.839627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840470 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.840471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.841330 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.841333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.842063 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.842117 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842121 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:16.842160 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 03:18:16.842194 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":29}
I20260927 03:18:16.842289 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.17.52.131:42637)
0927 03:18:16.839467 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.839469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.839477 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:16.839499 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.839499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.839514 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.839574 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:16.839708 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840411 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.841420 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.841991 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842066 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842078 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.842173 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.842209 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
I20260927 03:18:16.842619 24915 tablet_service.cc:1467] Processing CreateTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 (DEFAULT_TABLE table=test-table [id=a47fc1f0acc44956be0fe5eef8ed2f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:16.842700 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:34295 (local address 127.0.0.1:42598)
0927 03:18:16.839153 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.839164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:16.839214 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:16.839348 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:16.839348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:16.839350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:16.839401 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.839402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840434 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.840435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.841710 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.841712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.842470 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.842533 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842536 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:16.842591 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:18:16.842635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":98}
I20260927 03:18:16.842747 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4042b05e2049d0ab5796f95c54bbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:16.842916 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.17.52.130:34295)
0927 03:18:16.839222 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.839224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.839238 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:18:16.839267 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.839268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.839279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.839349 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:16.839482 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840351 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.841792 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.842386 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842470 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842483 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.842791 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.842845 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20260927 03:18:16.842996 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:41095 (local address 127.0.0.1:52482)
0927 03:18:16.839648 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.839659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:16.839705 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:16.839897 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:16.839897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:16.839899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:16.839944 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.839945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840690 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.840691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.841893 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:16.841895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.842665 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:16.842733 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842736 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:16.842873 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 03:18:16.842911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":81,"threads_started":1}
I20260927 03:18:16.843137 24779 tablet_service.cc:1467] Processing CreateTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 (DEFAULT_TABLE table=test-table [id=a47fc1f0acc44956be0fe5eef8ed2f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:16.843242 24779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4042b05e2049d0ab5796f95c54bbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:16.844029 25071 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Bootstrap starting.
I20260927 03:18:16.844355 25071 tablet_bootstrap.cc:654] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:16.844494 25072 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Bootstrap starting.
I20260927 03:18:16.844713 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.839367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.17.52.129:41095)
0927 03:18:16.839457 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.839459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.839759 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:18:16.839787 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.839788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.839799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.839854 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:16.840026 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.840604 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.841983 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.842568 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.842627 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.842637 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.844546 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.844626 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":47,"threads_started":1}
I20260927 03:18:16.844967 24642 tablet_service.cc:1467] Processing CreateTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 (DEFAULT_TABLE table=test-table [id=a47fc1f0acc44956be0fe5eef8ed2f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:16.845094 25071 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: No bootstrap required, opened a new log
I20260927 03:18:16.845099 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4042b05e2049d0ab5796f95c54bbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:16.845145 25071 ts_tablet_manager.cc:1397] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:16.845258 25071 raft_consensus.cc:348] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.845312 25071 raft_consensus.cc:374] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:16.845326 25071 raft_consensus.cc:729] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cd6500ef184de096d1f8843bc3e5e1, State: Initialized, Role: FOLLOWER
I20260927 03:18:16.845379 25071 consensus_queue.cc:260] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.845507 25071 ts_tablet_manager.cc:1428] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:16.846174 24981 heartbeater.cc:503] Master 127.17.52.188:46271 was elected leader, sending a full tablet report...
I20260927 03:18:16.846572 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.845685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:43403)
0927 03:18:16.845809 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.845847 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260927 03:18:16.846783 25072 tablet_bootstrap.cc:654] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:16.846952 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.846864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:58063)
0927 03:18:16.846902 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.846926 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:18:16.847107 25076 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: Bootstrap starting.
I20260927 03:18:16.847385 25072 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: No bootstrap required, opened a new log
I20260927 03:18:16.847437 25072 ts_tablet_manager.cc:1397] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:18:16.847484 25076 tablet_bootstrap.cc:654] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:16.847556 25072 raft_consensus.cc:348] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.847645 25072 raft_consensus.cc:374] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:16.847677 25072 raft_consensus.cc:729] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d16df85ae84ce6a39ef365f1f183f8, State: Initialized, Role: FOLLOWER
I20260927 03:18:16.847728 25072 consensus_queue.cc:260] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.847872 25072 ts_tablet_manager.cc:1428] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:16.848058 25076 tablet_bootstrap.cc:492] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: No bootstrap required, opened a new log
I20260927 03:18:16.848102 25076 ts_tablet_manager.cc:1397] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:16.848212 25076 raft_consensus.cc:348] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.848260 25076 raft_consensus.cc:374] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:16.848280 25076 raft_consensus.cc:729] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e1ad773ea544168a227898f0a88bb5, State: Initialized, Role: FOLLOWER
I20260927 03:18:16.848333 25076 consensus_queue.cc:260] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.848508 25076 ts_tablet_manager.cc:1428] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:16.848867 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.848657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:42097)
0927 03:18:16.848778 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.848831 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260927 03:18:16.849105 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.848656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:35019)
0927 03:18:16.849039 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.849072 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":80,"threads_started":1}
I20260927 03:18:16.849220 24845 heartbeater.cc:503] Master 127.17.52.188:46271 was elected leader, sending a full tablet report...
I20260927 03:18:16.849264 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.849144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:42827)
0927 03:18:16.849212 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.849236 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 03:18:16.849287 24708 heartbeater.cc:503] Master 127.17.52.188:46271 was elected leader, sending a full tablet report...
I20260927 03:18:16.849687 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.849478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:56851)
0927 03:18:16.849629 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.849659 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:18:16.926769 25074 raft_consensus.cc:482] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:16.926859 25074 raft_consensus.cc:504] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.927022 25074 leader_election.cc:288] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295), b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
I20260927 03:18:16.927255 24799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4042b05e2049d0ab5796f95c54bbd3" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" is_pre_election: true
I20260927 03:18:16.927304 24662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4042b05e2049d0ab5796f95c54bbd3" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e1ad773ea544168a227898f0a88bb5" is_pre_election: true
I20260927 03:18:16.927380 24662 raft_consensus.cc:2454] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 0.
I20260927 03:18:16.927389 24799 raft_consensus.cc:2454] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 0.
I20260927 03:18:16.927539 24869 leader_election.cc:302] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7d16df85ae84ce6a39ef365f1f183f8, f7cd6500ef184de096d1f8843bc3e5e1; no voters:
I20260927 03:18:16.927642 25074 raft_consensus.cc:2781] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:16.927686 25074 raft_consensus.cc:482] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:16.927717 25074 raft_consensus.cc:3037] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:16.928390 25074 raft_consensus.cc:504] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.928519 25074 leader_election.cc:288] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 election: Requested vote from peers b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295), b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095)
I20260927 03:18:16.928684 24799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4042b05e2049d0ab5796f95c54bbd3" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d16df85ae84ce6a39ef365f1f183f8"
I20260927 03:18:16.928714 24662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4042b05e2049d0ab5796f95c54bbd3" candidate_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e1ad773ea544168a227898f0a88bb5"
I20260927 03:18:16.928777 24799 raft_consensus.cc:3037] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:16.928788 24662 raft_consensus.cc:3037] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:16.929373 24799 raft_consensus.cc:2454] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 1.
I20260927 03:18:16.929415 24662 raft_consensus.cc:2454] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cd6500ef184de096d1f8843bc3e5e1 in term 1.
I20260927 03:18:16.929584 24869 leader_election.cc:302] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6e1ad773ea544168a227898f0a88bb5, f7cd6500ef184de096d1f8843bc3e5e1; no voters:
I20260927 03:18:16.929729 25074 raft_consensus.cc:2781] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:16.929926 25074 raft_consensus.cc:686] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 LEADER]: Becoming Leader. State: Replica: f7cd6500ef184de096d1f8843bc3e5e1, State: Running, Role: LEADER
I20260927 03:18:16.929997 25074 consensus_queue.cc:237] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } }
I20260927 03:18:16.930258 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.930126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:52917)
0927 03:18:16.930165 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.930196 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 03:18:16.930673 24529 catalog_manager.cc:5495] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7cd6500ef184de096d1f8843bc3e5e1" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42637 } health_report { overall_health: HEALTHY } } }
I20260927 03:18:16.957553 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.955371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.17.52.131:42637)
0927 03:18:16.955411 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:16.955413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:16.955422 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:16.955487 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:16.955488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:16.955496 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:16.955536 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:16.955707 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.956232 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.957017 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:16.957195 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:16.957244 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:16.957370 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:16.957395 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:16.957430 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:16.957484 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 03:18:16.957500 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:18:16.958549 24799 raft_consensus.cc:1261] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Refusing update from remote peer f7cd6500ef184de096d1f8843bc3e5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:16.958565 24662 raft_consensus.cc:1261] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Refusing update from remote peer f7cd6500ef184de096d1f8843bc3e5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:16.958788 25074 consensus_queue.cc:1035] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d16df85ae84ce6a39ef365f1f183f8" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:16.958891 25083 consensus_queue.cc:1035] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e1ad773ea544168a227898f0a88bb5" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:16.959673 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.959557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:51167)
0927 03:18:16.959595 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.959642 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:18:16.960024 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.959883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:55767)
0927 03:18:16.959954 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.959987 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 03:18:16.961194 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.961021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:56083)
0927 03:18:16.961130 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.961163 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 03:18:16.961620 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.961507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.130:47105)
0927 03:18:16.961549 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.961574 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:18:16.962100 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.961434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.131:49355)
0927 03:18:16.962050 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.962077 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601}
I20260927 03:18:16.962113 24528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:16.962224 24528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:16.963018 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.962885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:39093 (local address 127.17.52.129:39671)
0927 03:18:16.962960 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.962986 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 03:18:16.963645 24528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01a07a3a10bf43359a40cf357b1f4718: Sending DeleteTablet for 3 replicas of tablet bd4042b05e2049d0ab5796f95c54bbd3
I20260927 03:18:16.963887 24915 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:16 UTC) from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:38454
I20260927 03:18:16.963922 24914 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:16 UTC) from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:38454
I20260927 03:18:16.963980 24779 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:16 UTC) from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:42598
I20260927 03:18:16.964061 25095 tablet_replica.cc:330] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: stopping tablet replica
I20260927 03:18:16.964118 25095 raft_consensus.cc:2229] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:16.964172 25095 raft_consensus.cc:2258] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:16.964246 25096 tablet_replica.cc:330] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: stopping tablet replica
I20260927 03:18:16.964296 25096 raft_consensus.cc:2229] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:16.964337 25096 raft_consensus.cc:2258] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:16.964355 24642 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:16 UTC) from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:52482
I20260927 03:18:16.964505 24641 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4042b05e2049d0ab5796f95c54bbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:16 UTC) from {username='oryx', principal='oryx/127.17.52.188@KRBTEST.COM'} at 127.0.0.1:52482
I20260927 03:18:16.964519 25097 tablet_replica.cc:330] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5: stopping tablet replica
I20260927 03:18:16.964640 25096 ts_tablet_manager.cc:1905] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:16.964666 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24484
I20260927 03:18:16.964675 25097 raft_consensus.cc:2229] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:16.964831 25097 raft_consensus.cc:2258] T bd4042b05e2049d0ab5796f95c54bbd3 P b6e1ad773ea544168a227898f0a88bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:16.966419 25096 ts_tablet_manager.cc:1918] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:16.966477 25096 log.cc:1196] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/bd4042b05e2049d0ab5796f95c54bbd3
I20260927 03:18:16.966646 25096 ts_tablet_manager.cc:1939] T bd4042b05e2049d0ab5796f95c54bbd3 P b7d16df85ae84ce6a39ef365f1f183f8: Deleting consensus metadata
I20260927 03:18:16.966907 25095 ts_tablet_manager.cc:1905] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:16.966920 24514 catalog_manager.cc:4841] TS b7d16df85ae84ce6a39ef365f1f183f8 (127.17.52.130:34295): tablet bd4042b05e2049d0ab5796f95c54bbd3 (table test-table [id=a47fc1f0acc44956be0fe5eef8ed2f9d]) successfully deleted
I20260927 03:18:16.967998 25095 ts_tablet_manager.cc:1918] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:16.968044 25095 log.cc:1196] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/bd4042b05e2049d0ab5796f95c54bbd3
I20260927 03:18:16.968299 25095 ts_tablet_manager.cc:1939] T bd4042b05e2049d0ab5796f95c54bbd3 P f7cd6500ef184de096d1f8843bc3e5e1: Deleting consensus metadata
I20260927 03:18:16.969365 24515 catalog_manager.cc:4841] TS f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637): tablet bd4042b05e2049d0ab5796f95c54bbd3 (table test-table [id=a47fc1f0acc44956be0fe5eef8ed2f9d]) successfully deleted
W20260927 03:18:16.969475 24515 catalog_manager.cc:4808] TS f7cd6500ef184de096d1f8843bc3e5e1 (127.17.52.131:42637): delete failed for tablet bd4042b05e2049d0ab5796f95c54bbd3 because the tablet was not found. No further retry: Not found: Tablet not found: bd4042b05e2049d0ab5796f95c54bbd3
I20260927 03:18:16.971659 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:16.971565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:41095 (local address 127.0.0.1:52498)
0927 03:18:16.971613 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:16.971638 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:41095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:18:16.971763 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24716
W20260927 03:18:16.971807 24514 catalog_manager.cc:4584] TS b6e1ad773ea544168a227898f0a88bb5 (127.17.52.129:41095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd4042b05e2049d0ab5796f95c54bbd3: Network error: Client connection negotiation failed: client connection to 127.17.52.129:41095: connect: Connection refused (error 111)
I20260927 03:18:16.976698 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24852
I20260927 03:18:16.981738 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24201
I20260927 03:18:16.986301 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 24390
2026-09-27T03:18:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6197 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:17 dist-test-slave-7j43 krb5kdc[25100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:17 dist-test-slave-7j43 krb5kdc[25100](info): set up 2 sockets
Sep 27 03:18:17 dist-test-slave-7j43 krb5kdc[25100](info): commencing operation
krb5kdc: starting...
W20260927 03:18:18.029451 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab.
Entry for principal oryx/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab.
I20260927 03:18:18.143127 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:34615
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab
--principal=oryx/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:18.223796 25113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:18.223999 25113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:18.224022 25113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:18.225138 25113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:18.225175 25113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:18:18.225188 25113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:18.225201 25113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:18.225214 25113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:18.225225 25113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:18.226706 25113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:34615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.190.keytab
--principal=oryx/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:18.226897 25113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:18.227065 25113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:18.231823 25119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.231834 25113 server_base.cc:1034] running on GCE node
W20260927 03:18:18.231817 25121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.231817 25118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.232316 25113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:18.232671 25113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:18.233836 25113 hybrid_clock.cc:648] HybridClock initialized: now 1790479098233825 us; error 37 us; skew 500 ppm
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:18.236213 25113 init.cc:377] Logged in from keytab as oryx/127.17.52.190@KRBTEST.COM (short username oryx)
I20260927 03:18:18.236632 25113 webserver.cc:460] Webserver started at http://127.17.52.190:35887/ using document root <none> and password file <none>
I20260927 03:18:18.236850 25113 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:18.236892 25113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:18.236989 25113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:18.237993 25113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "595f0f86f5a74d829dc86d9ab29582d7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "eecac6d9adfe69dd2a2546a89e81ab30"
server_key_iv: "b2b8536ab8bd35c23d4ef6b221fccc6e"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.238286 25113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "595f0f86f5a74d829dc86d9ab29582d7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "eecac6d9adfe69dd2a2546a89e81ab30"
server_key_iv: "b2b8536ab8bd35c23d4ef6b221fccc6e"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.239522 25113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:18.240175 25128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.240350 25113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:18.240437 25113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "595f0f86f5a74d829dc86d9ab29582d7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "eecac6d9adfe69dd2a2546a89e81ab30"
server_key_iv: "b2b8536ab8bd35c23d4ef6b221fccc6e"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.240509 25113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:18.248343 25113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:18.250085 25113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:18.250229 25113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:18.254444 25113 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:34615
I20260927 03:18:18.254508 25190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:34615 every 8 connection(s)
I20260927 03:18:18.254817 25113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:18.255445 25191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.257727 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25113
I20260927 03:18:18.257828 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:18.258038 17618 external_mini_cluster.cc:1428] Setting key c4e0ecf387d443f7000f6c82b4ab811a
I20260927 03:18:18.258052 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Bootstrap starting.
I20260927 03:18:18.258790 25191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.259078 25191 log.cc:824] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:18.259735 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: No bootstrap required, opened a new log
I20260927 03:18:18.262084 25191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } }
I20260927 03:18:18.262252 25191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.262290 25191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595f0f86f5a74d829dc86d9ab29582d7, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.262408 25191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } }
I20260927 03:18:18.262477 25191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:18.262506 25191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:18.262549 25191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.263195 25191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } }
I20260927 03:18:18.263321 25191 leader_election.cc:302] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 595f0f86f5a74d829dc86d9ab29582d7; no voters:
I20260927 03:18:18.263507 25191 leader_election.cc:288] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:18.263552 25196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:18.263696 25196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [term 1 LEADER]: Becoming Leader. State: Replica: 595f0f86f5a74d829dc86d9ab29582d7, State: Running, Role: LEADER
I20260927 03:18:18.263782 25191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:18.263777 25196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } }
I20260927 03:18:18.264089 25196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 595f0f86f5a74d829dc86d9ab29582d7. Latest consensus state: current_term: 1 leader_uuid: "595f0f86f5a74d829dc86d9ab29582d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } } }
I20260927 03:18:18.264099 25197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "595f0f86f5a74d829dc86d9ab29582d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595f0f86f5a74d829dc86d9ab29582d7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34615 } } }
I20260927 03:18:18.264173 25196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:18.264174 25197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:18.264367 25200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:18.264853 25200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.266394 25200 catalog_manager.cc:1347] Generated new cluster ID: 72b7afcb75814bc39e9daa5cdaf5faf9
I20260927 03:18:18.266444 25200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:18.269822 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.258532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.17.52.190:34615)
0927 03:18:18.258692 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.258699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.258723 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:18:18.259217 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.259225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.259273 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.259383 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.260050 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.261491 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.262231 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.262842 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.262952 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.266289 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.266293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.266293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.266305 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.268218 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.268611 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.268614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.268615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.268724 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.268918 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.268919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.268920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.269220 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.269385 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.269460 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.269609 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260927 03:18:18.294111 25200 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:18.294898 25200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:18.298228 25200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Generated new TSK 0
I20260927 03:18:18.298424 25200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
I20260927 03:18:18.328375 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:34615
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:18.408084 25217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:18.408290 25217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:18.408314 25217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:18.409492 25217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:18.409536 25217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:18.409557 25217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:18:18.409600 25217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:18.411089 25217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:18.411338 25217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:18.411556 25217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:18.416399 25225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.416416 25223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.416414 25222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.416554 25217 server_base.cc:1034] running on GCE node
I20260927 03:18:18.416767 25217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:18.416946 25217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:18.418107 25217 hybrid_clock.cc:648] HybridClock initialized: now 1790479098418094 us; error 28 us; skew 500 ppm
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:18.420557 25217 init.cc:377] Logged in from keytab as oryx/127.17.52.129@KRBTEST.COM (short username oryx)
I20260927 03:18:18.421064 25217 webserver.cc:460] Webserver started at http://127.17.52.129:41261/ using document root <none> and password file <none>
I20260927 03:18:18.421278 25217 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:18.421344 25217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:18.421489 25217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:18.422513 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "21b594dc5ebc4508a30d62017bed6a73"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "0d8eef3349668942d2f880a80e502edc"
server_key_iv: "214d2c1a82b7f58d1e66bdf3018deec7"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.422890 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "21b594dc5ebc4508a30d62017bed6a73"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "0d8eef3349668942d2f880a80e502edc"
server_key_iv: "214d2c1a82b7f58d1e66bdf3018deec7"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.424140 25217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:18.424916 25232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.425081 25217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:18.425174 25217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "21b594dc5ebc4508a30d62017bed6a73"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "0d8eef3349668942d2f880a80e502edc"
server_key_iv: "214d2c1a82b7f58d1e66bdf3018deec7"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.425251 25217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:18.438375 25217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:18.440503 25217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:18.440671 25217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:18.441206 25217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:18.441259 25217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.441300 25217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:18.441327 25217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.448500 25217 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:36339
I20260927 03:18:18.448923 25217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:18.448979 25347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:36339 every 8 connection(s)
I20260927 03:18:18.451355 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.441544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:60717 (local address 127.17.52.190:34615)
0927 03:18:18.441598 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.441600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.441841 (+ 241us) server_negotiation.cc:407] Connection header received
0927 03:18:18.442548 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.442549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.442559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.442602 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.443570 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.444335 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.445995 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.446217 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.446311 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.448925 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.448926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.448926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.448933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.450265 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.450603 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.450603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.450604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.450654 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.450861 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.450862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.450862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.450958 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.451007 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.451212 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.451283 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:18:18.451469 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.441525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.129:60717)
0927 03:18:18.441688 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.441739 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.442446 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.442761 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.442766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.443158 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.443479 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.443485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.444406 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.444407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.445904 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.445907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.446331 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.446332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.446455 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.447342 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.447350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.448834 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.450331 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.450336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.450337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.450540 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.450705 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.450706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.450706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.450812 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.451007 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.451011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.451114 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.451191 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.451304 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260927 03:18:18.451936 25348 heartbeater.cc:347] Connected to a master server at 127.17.52.190:34615
I20260927 03:18:18.452028 25348 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:18.452188 25348 heartbeater.cc:511] Master 127.17.52.190:34615 requested a full tablet report, sending...
I20260927 03:18:18.452749 25145 ts_manager.cc:194] Registered new tserver with Master: 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339)
I20260927 03:18:18.453660 25145 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:60717
I20260927 03:18:18.454005 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25217
I20260927 03:18:18.454126 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:18.454347 17618 external_mini_cluster.cc:1428] Setting key 27a4c519634ca368f8d2aa82247a04f6
I20260927 03:18:18.458541 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.452990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:37363 (local address 127.17.52.190:34615)
0927 03:18:18.453040 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.453042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.453114 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:18:18.453182 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.453183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.453190 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.453232 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.453527 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.454127 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.455069 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.455266 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.455349 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.456174 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.456175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.456175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.456182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.457494 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.457885 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.457886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.457886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.457932 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.458129 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.458129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.458129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.458215 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.458284 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.458433 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.458487 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:18:18.458657 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.452921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.129:37363)
0927 03:18:18.453059 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.453070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.453129 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.453238 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.453238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.453418 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.453469 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.453470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.454244 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.454244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.454985 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.454987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.455351 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.455352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.455417 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.455826 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.455830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.456081 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.457570 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.457571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.457572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.457801 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.457989 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.457990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.457991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.458064 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.458279 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.458280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.458330 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.458412 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.458527 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":25,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
I20260927 03:18:18.477242 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:34615
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:18.556785 25357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:18.556986 25357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:18.557010 25357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:18.558097 25357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:18.558132 25357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:18.558152 25357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:18:18.558169 25357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:18.559561 25357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:18.559742 25357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:18.559914 25357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:18.564656 25365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.564692 25363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.564646 25362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.564908 25357 server_base.cc:1034] running on GCE node
I20260927 03:18:18.565135 25357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:18.565377 25357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:18.566540 25357 hybrid_clock.cc:648] HybridClock initialized: now 1790479098566519 us; error 33 us; skew 500 ppm
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:18.568969 25357 init.cc:377] Logged in from keytab as oryx/127.17.52.130@KRBTEST.COM (short username oryx)
I20260927 03:18:18.569375 25357 webserver.cc:460] Webserver started at http://127.17.52.130:34861/ using document root <none> and password file <none>
I20260927 03:18:18.569581 25357 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:18.569644 25357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:18.569787 25357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:18.570760 25357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "45806b928b96421195bdb84b82d42bb7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "d5c1c96363805f8a4b264b1e9f4f9bb0"
server_key_iv: "1003b6f032f6583b77e4e7adf270f538"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.571122 25357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "45806b928b96421195bdb84b82d42bb7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "d5c1c96363805f8a4b264b1e9f4f9bb0"
server_key_iv: "1003b6f032f6583b77e4e7adf270f538"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.572355 25357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:18.573058 25372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.573215 25357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.573311 25357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "45806b928b96421195bdb84b82d42bb7"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "d5c1c96363805f8a4b264b1e9f4f9bb0"
server_key_iv: "1003b6f032f6583b77e4e7adf270f538"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.573385 25357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:18.600554 25357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:18.602875 25357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:18.603037 25357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:18.603538 25357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:18.603596 25357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.603641 25357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:18.603683 25357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.610929 25357 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:35825
I20260927 03:18:18.611133 25487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:35825 every 8 connection(s)
I20260927 03:18:18.611371 25357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:18.611513 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25357
I20260927 03:18:18.611624 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:18.611857 17618 external_mini_cluster.cc:1428] Setting key ffebe34949aa75a0610c6134b565b19a
I20260927 03:18:18.613730 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.604091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:57875 (local address 127.17.52.190:34615)
0927 03:18:18.604132 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.604135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.604416 (+ 281us) server_negotiation.cc:407] Connection header received
0927 03:18:18.604995 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.604996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.605009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.605075 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.605667 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.606489 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.607574 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.607919 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.607993 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.610591 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.610592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.610592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.610600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.612246 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.612732 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.612733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.612734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.612801 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.613060 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.613061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.613061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.613185 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.613253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.613556 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.613637 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:18:18.613909 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.604088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.130:57875)
0927 03:18:18.604285 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.604320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.604885 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.605162 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.605166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.605398 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.605574 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.605577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.606577 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.606578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.607492 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.607494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.608028 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.608029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.608204 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.608816 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.608828 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.610501 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.612327 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.612333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.612335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.612656 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.612870 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.612871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.612871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.612981 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.613256 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.613262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.613416 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.613521 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.613681 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260927 03:18:18.614459 25488 heartbeater.cc:347] Connected to a master server at 127.17.52.190:34615
I20260927 03:18:18.614538 25488 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:18.614727 25488 heartbeater.cc:511] Master 127.17.52.190:34615 requested a full tablet report, sending...
I20260927 03:18:18.615115 25145 ts_manager.cc:194] Registered new tserver with Master: 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825)
I20260927 03:18:18.615608 25145 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:57875
WARNING: no policy specified for oryx/127.17.52.131@KRBTEST.COM; defaulting to no policy
I20260927 03:18:18.621049 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.615498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:34715 (local address 127.17.52.190:34615)
0927 03:18:18.615536 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.615538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.615653 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:18:18.615846 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.615847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.615856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.615902 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.616213 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.616776 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.617473 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.617669 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.617734 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.618613 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.618615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.618615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.618622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.619878 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.620213 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.620214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.620214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.620276 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.620461 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.620462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.620462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.620663 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.620725 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.620880 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.620959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:18.621062 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.615453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.130:34715)
0927 03:18:18.615586 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.615596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.615703 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.615906 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.615907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.616079 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.616142 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.616143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.616885 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.616886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.617384 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.617385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.617743 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.617744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.617808 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.618265 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.618269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.618530 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.619959 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.619960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.619960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.620132 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.620337 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.620337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.620338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.620395 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.620728 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.620729 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.620774 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.620855 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.620957 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
I20260927 03:18:18.635288 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:34615
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:18.715059 25497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:18.715287 25497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:18.715328 25497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:18.716465 25497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:18.716528 25497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:18.716563 25497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:18:18.716610 25497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:18.718070 25497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.52.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:18.718317 25497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:18.718529 25497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:18.723230 25502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:18.723263 25503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.723390 25497 server_base.cc:1034] running on GCE node
W20260927 03:18:18.723453 25505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:18.723652 25497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:18.723879 25497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:18.725030 25497 hybrid_clock.cc:648] HybridClock initialized: now 1790479098725016 us; error 30 us; skew 500 ppm
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:18.727262 25497 init.cc:377] Logged in from keytab as oryx/127.17.52.131@KRBTEST.COM (short username oryx)
I20260927 03:18:18.727681 25497 webserver.cc:460] Webserver started at http://127.17.52.131:45051/ using document root <none> and password file <none>
I20260927 03:18:18.727932 25497 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:18.727993 25497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:18.728120 25497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:18.729221 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "7ccc307a4f68635b77a0ddc20edc441e"
server_key_iv: "cdc8b884d8411c8371780bbd8d72693f"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.729568 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "7ccc307a4f68635b77a0ddc20edc441e"
server_key_iv: "cdc8b884d8411c8371780bbd8d72693f"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.730813 25497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:18.731469 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.731624 25497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:18.731719 25497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
format_stamp: "Formatted at 2026-09-27 03:18:18 on dist-test-slave-7j43"
server_key: "7ccc307a4f68635b77a0ddc20edc441e"
server_key_iv: "cdc8b884d8411c8371780bbd8d72693f"
server_key_version: "encryptionkey@0"
I20260927 03:18:18.731791 25497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:18.745390 25497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:18.747171 25497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:18.747315 25497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:18.747857 25497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:18.747910 25497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.747941 25497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:18.747968 25497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.755218 25497 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:34437
I20260927 03:18:18.755300 25627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:34437 every 8 connection(s)
I20260927 03:18:18.755642 25497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:18.757537 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.748162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:60029 (local address 127.17.52.190:34615)
0927 03:18:18.748218 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.748221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.748415 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:18:18.748968 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.748970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.748982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.749038 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.749680 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.750201 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.751277 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.751524 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.751604 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.754570 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.754572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.754572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.754581 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.756175 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.756595 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.756597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.756597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.756655 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.756919 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.756920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.756920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.757038 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.757110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.757365 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.757442 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:18:18.757735 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.748172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.131:60029)
0927 03:18:18.748307 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.748339 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.748873 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.749106 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.749110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.749407 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.749593 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.749598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.750304 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.750305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.751194 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.751197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.751612 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.751613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.751736 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.752215 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.752225 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.754471 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.756271 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.756276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.756279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.756489 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.756740 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.756741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.756741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.756853 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.757101 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.757107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.757240 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.757346 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.757502 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260927 03:18:18.758173 25628 heartbeater.cc:347] Connected to a master server at 127.17.52.190:34615
I20260927 03:18:18.758255 25628 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:18.758445 25628 heartbeater.cc:511] Master 127.17.52.190:34615 requested a full tablet report, sending...
I20260927 03:18:18.758824 25144 ts_manager.cc:194] Registered new tserver with Master: 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:18.759313 25144 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:60029
I20260927 03:18:18.759631 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25497
I20260927 03:18:18.759763 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:18.760008 17618 external_mini_cluster.cc:1428] Setting key 56e61a50654249715d8af7e824f66e34
I20260927 03:18:18.760885 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:18:18.765352 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.759195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:55857 (local address 127.17.52.190:34615)
0927 03:18:18.759234 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.759236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.759332 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:18:18.759453 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.759454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.759465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.759532 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.759977 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.760732 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.761582 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.761890 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.761963 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.762755 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.762756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.762757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.762763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.764164 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.764553 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.764554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.764555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.764619 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.764796 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.764797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.764797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.764932 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.765009 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.765178 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.765247 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:18:18.765363 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.759140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.17.52.131:55857)
0927 03:18:18.759272 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.759281 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.759346 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.759575 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.759577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.759798 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.759878 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.759880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.760821 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.760822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.761510 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.761512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.761960 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.761960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.762004 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.762387 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.762392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.762652 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.764247 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.764248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.764249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.764463 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.764677 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.764678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.764678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.764730 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.765029 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.765030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.765087 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.765152 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.765249 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:18:18.781386 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:18.776623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.17.52.190:34615)
0927 03:18:18.776662 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.776664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.776671 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:18.776736 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.776736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.776759 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.776825 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.777311 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.777762 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.778416 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.778584 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.778629 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.781305 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42546: BlockingRecv error: recv got EOF from 127.0.0.1:42546 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.787941 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.782156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.17.52.190:34615)
0927 03:18:18.782217 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.782219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.782305 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:18:18.782377 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.782377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.782387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.782440 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.782873 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.783322 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.783971 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.784158 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.784222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.786041 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.786043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.786043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.786048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.787120 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.787406 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.787407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.787407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.787450 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.787591 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.787592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.787592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.787696 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.787764 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.787840 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.787889 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:18:18.789371 25144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:18.790874 25144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:18.796681 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.792853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34437 (local address 127.0.0.1:44272)
0927 03:18:18.793018 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.793055 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.793116 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.793422 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.793425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.793427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:18.793490 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.793491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794427 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.794428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.795582 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.795585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796356 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.796432 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796445 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.796540 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.796608 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260927 03:18:18.796705 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.793005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.17.52.131:34437)
0927 03:18:18.793266 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.793270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.793278 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:18.793316 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.793318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.793333 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.793422 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:18.793572 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794345 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.795673 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796279 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796356 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796429 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.796557 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.796619 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260927 03:18:18.796846 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.792852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36339 (local address 127.0.0.1:35196)
0927 03:18:18.793207 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.793216 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.793251 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.793450 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.793452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.793454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:18.793623 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.793625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794895 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.794896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.795739 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.795741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796618 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.796689 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796695 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.796727 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.796766 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260927 03:18:18.797152 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.793095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.17.52.129:36339)
0927 03:18:18.793228 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.793234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.793249 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:18:18.793335 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.793337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.793350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.793708 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:18.793767 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794799 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.795823 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796488 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796919 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796995 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.797018 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.797064 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260927 03:18:18.797159 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.792864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35825 (local address 127.0.0.1:32910)
0927 03:18:18.793283 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.793293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.793334 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.793529 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.793530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.793531 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:18.793583 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.793585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794841 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.794842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.795932 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.795934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796918 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.796984 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796988 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.797058 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.797090 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":24,"thread_start_us":48,"threads_started":1}
I20260927 03:18:18.797343 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.793107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.17.52.130:35825)
0927 03:18:18.793384 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.793389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.793398 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:18.793432 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.793434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.793448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.793520 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:18.793667 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.794748 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796008 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.796748 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.796818 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.796886 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.797182 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.797224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":83,"threads_started":1}
I20260927 03:18:18.797690 25562 tablet_service.cc:1467] Processing CreateTablet for tablet 5f336bf65be8412b877f5749a25287fe (DEFAULT_TABLE table=test-table [id=b89c2151f1f547bc963bc710ecad472d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.797706 25282 tablet_service.cc:1467] Processing CreateTablet for tablet 5f336bf65be8412b877f5749a25287fe (DEFAULT_TABLE table=test-table [id=b89c2151f1f547bc963bc710ecad472d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.797895 25422 tablet_service.cc:1467] Processing CreateTablet for tablet 5f336bf65be8412b877f5749a25287fe (DEFAULT_TABLE table=test-table [id=b89c2151f1f547bc963bc710ecad472d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.798062 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f336bf65be8412b877f5749a25287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.798075 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f336bf65be8412b877f5749a25287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.798235 25422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f336bf65be8412b877f5749a25287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.801159 25653 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Bootstrap starting.
I20260927 03:18:18.801803 25653 tablet_bootstrap.cc:654] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.801875 25654 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Bootstrap starting.
I20260927 03:18:18.802075 25655 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Bootstrap starting.
I20260927 03:18:18.802317 25653 log.cc:824] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:18.802479 25654 tablet_bootstrap.cc:654] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.802693 25655 tablet_bootstrap.cc:654] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.802733 25654 log.cc:824] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:18.803033 25655 log.cc:824] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:18.803145 25653 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: No bootstrap required, opened a new log
I20260927 03:18:18.803215 25653 ts_tablet_manager.cc:1397] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:18.803464 25654 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: No bootstrap required, opened a new log
I20260927 03:18:18.803530 25654 ts_tablet_manager.cc:1397] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:18.803782 25655 tablet_bootstrap.cc:492] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: No bootstrap required, opened a new log
I20260927 03:18:18.803858 25655 ts_tablet_manager.cc:1397] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:18.805233 25655 raft_consensus.cc:348] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805203 25653 raft_consensus.cc:348] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805380 25653 raft_consensus.cc:374] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.805382 25655 raft_consensus.cc:374] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.805426 25655 raft_consensus.cc:729] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7403ea84feaa44f4b1941e5b389cdb0b, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.805426 25653 raft_consensus.cc:729] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.805444 25654 raft_consensus.cc:348] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805531 25655 consensus_queue.cc:260] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805577 25654 raft_consensus.cc:374] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.805555 25653 consensus_queue.cc:260] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805608 25654 raft_consensus.cc:729] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.805706 25654 consensus_queue.cc:260] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.805778 25655 ts_tablet_manager.cc:1428] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:18:18.805897 25628 heartbeater.cc:503] Master 127.17.52.190:34615 was elected leader, sending a full tablet report...
I20260927 03:18:18.805949 25654 ts_tablet_manager.cc:1428] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:18:18.806080 25348 heartbeater.cc:503] Master 127.17.52.190:34615 was elected leader, sending a full tablet report...
I20260927 03:18:18.805845 25653 ts_tablet_manager.cc:1428] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:18:18.805929 25488 heartbeater.cc:503] Master 127.17.52.190:34615 was elected leader, sending a full tablet report...
W20260927 03:18:18.862152 25489 tablet.cc:2369] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:18:18.883976 25660 raft_consensus.cc:482] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:18.884094 25660 raft_consensus.cc:504] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.884501 25660 leader_election.cc:288] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437), 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339)
I20260927 03:18:18.888296 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.884776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:49389 (local address 127.17.52.131:34437)
0927 03:18:18.884836 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.884838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.884871 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:18:18.884901 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.884902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.884912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.884970 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:18.885135 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.885908 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.887395 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.887949 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.888024 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.888033 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.888191 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.888228 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:18:18.888306 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.884831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:56861 (local address 127.17.52.129:36339)
0927 03:18:18.884875 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.884877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.885035 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:18:18.885064 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.885065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.885072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.885129 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:18.885255 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.885936 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.887395 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.887949 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.888024 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.888033 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.888191 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.888228 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:18.888329 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.884704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36339 (local address 127.17.52.130:56861)
0927 03:18:18.884931 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.884938 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.885000 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.885133 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.885134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.885135 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:18.885194 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.885195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.886005 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.886006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.887307 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.887311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.888030 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.888031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.888113 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.888117 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.888174 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.888219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":136,"thread_start_us":103,"threads_started":1}
I20260927 03:18:18.888335 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.884694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34437 (local address 127.17.52.130:49389)
0927 03:18:18.884752 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.884764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.884832 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.884986 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.884987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.884989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:18.885061 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.885067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.886002 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.886003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.887310 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.887311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.888030 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.888031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.888113 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.888117 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.888174 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.888219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":103}
I20260927 03:18:18.888861 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f336bf65be8412b877f5749a25287fe" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73" is_pre_election: true
I20260927 03:18:18.888864 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f336bf65be8412b877f5749a25287fe" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" is_pre_election: true
I20260927 03:18:18.889030 25302 raft_consensus.cc:2454] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:18.889035 25582 raft_consensus.cc:2454] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:18.889535 25374 leader_election.cc:302] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:18.889678 25660 raft_consensus.cc:2781] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:18.889736 25660 raft_consensus.cc:482] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:18.889766 25660 raft_consensus.cc:3037] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.890447 25660 raft_consensus.cc:504] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.890589 25660 leader_election.cc:288] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Requested vote from peers 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437), 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339)
I20260927 03:18:18.890743 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f336bf65be8412b877f5749a25287fe" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73"
I20260927 03:18:18.890734 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f336bf65be8412b877f5749a25287fe" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
I20260927 03:18:18.890825 25302 raft_consensus.cc:3037] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.890823 25582 raft_consensus.cc:3037] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.891487 25302 raft_consensus.cc:2454] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:18.891499 25582 raft_consensus.cc:2454] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:18.891685 25375 leader_election.cc:302] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45806b928b96421195bdb84b82d42bb7, 7403ea84feaa44f4b1941e5b389cdb0b; no voters:
I20260927 03:18:18.891801 25660 raft_consensus.cc:2781] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:18.891989 25660 raft_consensus.cc:686] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Running, Role: LEADER
I20260927 03:18:18.892086 25660 consensus_queue.cc:237] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } }
I20260927 03:18:18.892887 25145 catalog_manager.cc:5495] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45806b928b96421195bdb84b82d42bb7 (127.17.52.130). New cstate: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } health_report { overall_health: HEALTHY } } }
I20260927 03:18:18.916254 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.914079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.17.52.130:35825)
0927 03:18:18.914132 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.914134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.914145 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:18.914186 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.914187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.914204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.914265 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:18.914386 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.914841 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.915560 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.915770 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.915825 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.915947 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:18.916039 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:18.916094 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.916137 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.916153 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:18:18.917924 25302 raft_consensus.cc:1261] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:18.917927 25582 raft_consensus.cc:1261] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:18.918272 25663 consensus_queue.cc:1035] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:18.918392 25660 consensus_queue.cc:1035] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:18.920626 25666 mvcc.cc:205] Tried to move back new op lower bound from 7333802389165355008 to 7333802389062520832. Current Snapshot: MvccSnapshot[applied={T|T < 7333802389165355008}]
I20260927 03:18:18.920774 25668 mvcc.cc:205] Tried to move back new op lower bound from 7333802389165355008 to 7333802389062520832. Current Snapshot: MvccSnapshot[applied={T|T < 7333802389165355008}]
I20260927 03:18:18.922827 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:18.922923 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:18.923593 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Sending DeleteTablet for 3 replicas of tablet 5f336bf65be8412b877f5749a25287fe
I20260927 03:18:18.923905 25562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f336bf65be8412b877f5749a25287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:18 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260927 03:18:18.924009 25282 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f336bf65be8412b877f5749a25287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:18 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:35196
I20260927 03:18:18.924108 25422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f336bf65be8412b877f5749a25287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:18 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260927 03:18:18.924120 25676 tablet_replica.cc:330] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: stopping tablet replica
I20260927 03:18:18.924213 25676 raft_consensus.cc:2229] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:18.924311 25676 raft_consensus.cc:2258] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:18.924275 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:18.924403 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:18.924679 25678 tablet_replica.cc:330] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: stopping tablet replica
I20260927 03:18:18.924856 25678 raft_consensus.cc:2229] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:18.924928 25676 ts_tablet_manager.cc:1905] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:18.925076 25678 raft_consensus.cc:2258] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:18.925597 25678 ts_tablet_manager.cc:1905] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:18.925784 25562 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac730bf107d4ee58f321482680ddc28 (DEFAULT_TABLE table=test-table [id=a7a7dfccac5f45d89708de3371613451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.925895 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac730bf107d4ee58f321482680ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.926875 25676 ts_tablet_manager.cc:1918] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:18.926947 25676 log.cc:1196] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/5f336bf65be8412b877f5749a25287fe
I20260927 03:18:18.927240 25655 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Bootstrap starting.
I20260927 03:18:18.927250 25676 ts_tablet_manager.cc:1939] T 5f336bf65be8412b877f5749a25287fe P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting consensus metadata
I20260927 03:18:18.927505 25422 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac730bf107d4ee58f321482680ddc28 (DEFAULT_TABLE table=test-table [id=a7a7dfccac5f45d89708de3371613451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.927595 25422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac730bf107d4ee58f321482680ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.927664 25655 tablet_bootstrap.cc:654] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.927662 25131 catalog_manager.cc:4841] TS 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437): tablet 5f336bf65be8412b877f5749a25287fe (table test-table [id=b89c2151f1f547bc963bc710ecad472d]) successfully deleted
I20260927 03:18:18.928249 25655 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: No bootstrap required, opened a new log
I20260927 03:18:18.928298 25655 ts_tablet_manager.cc:1397] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:18.928269 25678 ts_tablet_manager.cc:1918] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:18.928436 25655 raft_consensus.cc:348] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.928468 25678 log.cc:1196] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/5f336bf65be8412b877f5749a25287fe
I20260927 03:18:18.928503 25655 raft_consensus.cc:374] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.928529 25655 raft_consensus.cc:729] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7403ea84feaa44f4b1941e5b389cdb0b, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.928594 25655 consensus_queue.cc:260] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.928680 25655 ts_tablet_manager.cc:1428] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.928926 25678 ts_tablet_manager.cc:1939] T 5f336bf65be8412b877f5749a25287fe P 45806b928b96421195bdb84b82d42bb7: Deleting consensus metadata
I20260927 03:18:18.928926 25653 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Bootstrap starting.
I20260927 03:18:18.929471 25653 tablet_bootstrap.cc:654] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.929868 25132 catalog_manager.cc:4841] TS 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825): tablet 5f336bf65be8412b877f5749a25287fe (table test-table [id=b89c2151f1f547bc963bc710ecad472d]) successfully deleted
I20260927 03:18:18.930249 25282 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac730bf107d4ee58f321482680ddc28 (DEFAULT_TABLE table=test-table [id=a7a7dfccac5f45d89708de3371613451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:18.930359 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac730bf107d4ee58f321482680ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.931248 25654 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Bootstrap starting.
I20260927 03:18:18.931387 25653 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: No bootstrap required, opened a new log
I20260927 03:18:18.931427 25653 ts_tablet_manager.cc:1397] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:18:18.931541 25653 raft_consensus.cc:348] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.931582 25653 raft_consensus.cc:374] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.931594 25653 raft_consensus.cc:729] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.931663 25653 consensus_queue.cc:260] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.931770 25653 ts_tablet_manager.cc:1428] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:18.931876 25654 tablet_bootstrap.cc:654] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.932277 25677 tablet_replica.cc:330] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: stopping tablet replica
I20260927 03:18:18.932381 25677 raft_consensus.cc:2229] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:18.932507 25677 raft_consensus.cc:2258] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:18.932642 25654 tablet_bootstrap.cc:492] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: No bootstrap required, opened a new log
I20260927 03:18:18.932683 25654 ts_tablet_manager.cc:1397] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:18.932827 25654 raft_consensus.cc:348] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.932894 25677 ts_tablet_manager.cc:1905] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:18.932885 25654 raft_consensus.cc:374] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.933001 25654 raft_consensus.cc:729] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.933055 25654 consensus_queue.cc:260] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.933169 25654 ts_tablet_manager.cc:1428] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:18.934374 25677 ts_tablet_manager.cc:1918] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:18.934478 25677 log.cc:1196] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/5f336bf65be8412b877f5749a25287fe
I20260927 03:18:18.934681 25677 ts_tablet_manager.cc:1939] T 5f336bf65be8412b877f5749a25287fe P 21b594dc5ebc4508a30d62017bed6a73: Deleting consensus metadata
I20260927 03:18:18.935055 25130 catalog_manager.cc:4841] TS 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339): tablet 5f336bf65be8412b877f5749a25287fe (table test-table [id=b89c2151f1f547bc963bc710ecad472d]) successfully deleted
I20260927 03:18:18.944887 25671 raft_consensus.cc:482] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:18.944939 25671 raft_consensus.cc:504] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.945092 25671 leader_election.cc:288] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:18.945240 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac730bf107d4ee58f321482680ddc28" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73" is_pre_election: true
I20260927 03:18:18.945231 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac730bf107d4ee58f321482680ddc28" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" is_pre_election: true
I20260927 03:18:18.945312 25302 raft_consensus.cc:2454] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:18.945318 25582 raft_consensus.cc:2454] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:18.945447 25374 leader_election.cc:302] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:18.945535 25671 raft_consensus.cc:2781] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:18.945570 25671 raft_consensus.cc:482] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:18.945590 25671 raft_consensus.cc:3037] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.946312 25671 raft_consensus.cc:504] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.946436 25671 leader_election.cc:288] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Requested vote from peers 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:18.946563 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac730bf107d4ee58f321482680ddc28" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
I20260927 03:18:18.946604 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac730bf107d4ee58f321482680ddc28" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73"
I20260927 03:18:18.946633 25582 raft_consensus.cc:3037] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.946658 25302 raft_consensus.cc:3037] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:18.947281 25302 raft_consensus.cc:2454] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:18.947317 25582 raft_consensus.cc:2454] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:18.947449 25374 leader_election.cc:302] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:18.947541 25671 raft_consensus.cc:2781] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:18.947587 25671 raft_consensus.cc:686] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Running, Role: LEADER
I20260927 03:18:18.947630 25671 consensus_queue.cc:237] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.948102 25144 catalog_manager.cc:5495] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45806b928b96421195bdb84b82d42bb7 (127.17.52.130). New cstate: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } health_report { overall_health: UNKNOWN } } }
W20260927 03:18:18.949657 25349 tablet.cc:2369] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:18:18 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479098, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.190@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:18.975972 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.969252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.17.52.190:34615)
0927 03:18:18.969310 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:18.969324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:18.969441 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:18:18.969519 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:18.969520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:18.969529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:18.969582 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:18.970106 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.970589 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.971288 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.971467 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.971545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.973576 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:18.973577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:18.973577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:18.973584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:18.974832 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.975195 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.975196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.975196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.975249 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:18.975445 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:18.975446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:18.975446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:18.975554 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:18.975639 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:18.975784 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:18:18.975866 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:18:18.976060 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:18.969210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34615 (local address 127.0.0.1:42560)
0927 03:18:18.969366 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:18.969376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:18.969460 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:18.969601 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:18.969602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:18.969966 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:18.970021 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.970023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.970665 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.970666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.971193 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:18.971194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:18.971555 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:18.971556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:18.971618 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:18.972153 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:18.972157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:18.973492 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:18.974923 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.974930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.974932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.975118 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.975326 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:18.975327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:18.975327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:18.975387 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:18.975629 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:18.975636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:18.975681 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:18:18.975768 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:18:18.975936 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":48,"thread_start_us":78,"threads_started":1}
I20260927 03:18:18.977919 25144 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:42560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:18:18.978121 25144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:18.979383 25282 tablet_service.cc:1467] Processing CreateTablet for tablet be27e30af177469496f7e52248a10233 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fe9022efd49478aa138fe6546782225]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:18:18.979426 25422 tablet_service.cc:1467] Processing CreateTablet for tablet be27e30af177469496f7e52248a10233 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fe9022efd49478aa138fe6546782225]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:18:18.979427 25562 tablet_service.cc:1467] Processing CreateTablet for tablet be27e30af177469496f7e52248a10233 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fe9022efd49478aa138fe6546782225]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:18:18.979524 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27e30af177469496f7e52248a10233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.979557 25422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27e30af177469496f7e52248a10233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.979558 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27e30af177469496f7e52248a10233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:18.980984 25653 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: Bootstrap starting.
I20260927 03:18:18.981052 25655 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: Bootstrap starting.
I20260927 03:18:18.981364 25653 tablet_bootstrap.cc:654] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.981456 25655 tablet_bootstrap.cc:654] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.981762 25654 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: Bootstrap starting.
I20260927 03:18:18.981912 25653 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: No bootstrap required, opened a new log
I20260927 03:18:18.981943 25653 ts_tablet_manager.cc:1397] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:18.982069 25653 raft_consensus.cc:348] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.982126 25653 raft_consensus.cc:374] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.982120 25654 tablet_bootstrap.cc:654] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:18.982141 25653 raft_consensus.cc:729] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.982191 25653 consensus_queue.cc:260] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.982300 25653 ts_tablet_manager.cc:1428] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:18.982316 25671 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:18.982394 25671 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:18.983090 25654 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: No bootstrap required, opened a new log
I20260927 03:18:18.983141 25654 ts_tablet_manager.cc:1397] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:18.983266 25654 raft_consensus.cc:348] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.983335 25654 raft_consensus.cc:374] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.983356 25654 raft_consensus.cc:729] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.983392 25654 consensus_queue.cc:260] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.983475 25654 ts_tablet_manager.cc:1428] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.983538 25661 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:18.983633 25661 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:18.984040 25655 tablet_bootstrap.cc:492] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: No bootstrap required, opened a new log
I20260927 03:18:18.984090 25655 ts_tablet_manager.cc:1397] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:18:18.984225 25655 raft_consensus.cc:348] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.984305 25655 raft_consensus.cc:374] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:18.984336 25655 raft_consensus.cc:729] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7403ea84feaa44f4b1941e5b389cdb0b, State: Initialized, Role: FOLLOWER
I20260927 03:18:18.984385 25655 consensus_queue.cc:260] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:18.984468 25655 ts_tablet_manager.cc:1428] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:18.984694 25659 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:18.984817 25659 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:18:19.006224 25629 tablet.cc:2369] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:18:19.091643 25671 raft_consensus.cc:482] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:19.091737 25671 raft_consensus.cc:504] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.091934 25671 leader_election.cc:288] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.092144 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27e30af177469496f7e52248a10233" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73" is_pre_election: true
I20260927 03:18:19.092168 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27e30af177469496f7e52248a10233" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" is_pre_election: true
I20260927 03:18:19.092257 25582 raft_consensus.cc:2454] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:19.092257 25302 raft_consensus.cc:2454] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 0.
I20260927 03:18:19.092412 25374 leader_election.cc:302] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:19.092504 25671 raft_consensus.cc:2781] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:19.092545 25671 raft_consensus.cc:482] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:19.092597 25671 raft_consensus.cc:3037] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.093220 25671 raft_consensus.cc:504] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.093346 25671 leader_election.cc:288] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Requested vote from peers 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.093482 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27e30af177469496f7e52248a10233" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b594dc5ebc4508a30d62017bed6a73"
I20260927 03:18:19.093493 25582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27e30af177469496f7e52248a10233" candidate_uuid: "45806b928b96421195bdb84b82d42bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
I20260927 03:18:19.093571 25582 raft_consensus.cc:3037] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.093575 25302 raft_consensus.cc:3037] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.094122 25582 raft_consensus.cc:2454] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:19.094182 25302 raft_consensus.cc:2454] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45806b928b96421195bdb84b82d42bb7 in term 1.
I20260927 03:18:19.094306 25375 leader_election.cc:302] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45806b928b96421195bdb84b82d42bb7, 7403ea84feaa44f4b1941e5b389cdb0b; no voters:
I20260927 03:18:19.094395 25671 raft_consensus.cc:2781] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:19.094460 25671 raft_consensus.cc:686] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Running, Role: LEADER
I20260927 03:18:19.094517 25671 consensus_queue.cc:237] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.094766 25660 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: TxnStatusTablet state changed. Reason: New leader 45806b928b96421195bdb84b82d42bb7. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.094851 25660 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:18:19.095047 25693 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:18:19.095135 25693 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:18:19.095090 25145 catalog_manager.cc:5495] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45806b928b96421195bdb84b82d42bb7 (127.17.52.130). New cstate: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:19.129504 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.126763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.17.52.130:35825)
0927 03:18:19.126823 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.126826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.126860 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:18:19.126941 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.126943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.126955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.127023 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:19.127151 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.127692 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.128867 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.129042 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.129095 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.129277 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.129302 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.129349 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.129432 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.129449 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:18:19.129556 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.126714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35825 (local address 127.0.0.1:32918)
0927 03:18:19.126816 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.126827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.126880 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.127033 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.127034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.127036 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:18:19.127086 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.127088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.127824 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.127825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.128788 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.128792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.129113 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.129113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.129180 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.129196 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.129369 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.129376 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.129430 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.129476 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 03:18:19.130548 25302 raft_consensus.cc:1261] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.130542 25582 raft_consensus.cc:1261] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.130752 25671 consensus_queue.cc:1035] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.130858 25660 consensus_queue.cc:1035] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.131382 25661 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: TxnStatusTablet state changed. Reason: New leader 45806b928b96421195bdb84b82d42bb7. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.131485 25661 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:19.131574 25659 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: TxnStatusTablet state changed. Reason: New leader 45806b928b96421195bdb84b82d42bb7. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.131672 25659 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:19.131815 25660 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.131904 25660 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:18:19.132531 25661 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.132644 25661 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 21b594dc5ebc4508a30d62017bed6a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:19.133606 25663 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.133690 25663 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 45806b928b96421195bdb84b82d42bb7: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:18:19.134071 25659 tablet_replica.cc:439] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45806b928b96421195bdb84b82d42bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } } }
I20260927 03:18:19.134150 25659 tablet_replica.cc:442] T be27e30af177469496f7e52248a10233 P 7403ea84feaa44f4b1941e5b389cdb0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:18:19.138247 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.135752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35825 (local address 127.17.52.130:53011)
0927 03:18:19.135880 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.135890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.135944 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.136059 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.136060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.136061 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:18:19.136102 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.136103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.136925 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.136926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.137475 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.137476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.137902 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.137903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.137948 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.137960 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.138107 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.138108 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.138141 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.138171 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260927 03:18:19.138249 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.135794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:53011 (local address 127.17.52.130:35825)
0927 03:18:19.135832 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.135834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.135947 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:18:19.135985 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.135986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.135995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.136058 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:19.136136 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.136852 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.137540 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.137829 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.137889 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.138022 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.138051 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.138103 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.138143 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.138160 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:19.140151 25582 raft_consensus.cc:1261] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.140170 25302 raft_consensus.cc:1261] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Refusing update from remote peer 45806b928b96421195bdb84b82d42bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.140340 25671 consensus_queue.cc:1035] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.140409 25660 consensus_queue.cc:1035] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.153009 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.153127 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.153681 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Sending DeleteTablet for 3 replicas of tablet 8ac730bf107d4ee58f321482680ddc28
I20260927 03:18:19.153955 25421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac730bf107d4ee58f321482680ddc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260927 03:18:19.154049 25282 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac730bf107d4ee58f321482680ddc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:35196
I20260927 03:18:19.154064 25678 tablet_replica.cc:330] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: stopping tablet replica
I20260927 03:18:19.154127 25678 raft_consensus.cc:2229] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:19.154191 25677 tablet_replica.cc:330] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: stopping tablet replica
I20260927 03:18:19.154197 25678 raft_consensus.cc:2258] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.154240 25677 raft_consensus.cc:2229] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.154284 25677 raft_consensus.cc:2258] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.154484 25678 ts_tablet_manager.cc:1905] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.154575 25562 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac730bf107d4ee58f321482680ddc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260927 03:18:19.154649 25676 tablet_replica.cc:330] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: stopping tablet replica
I20260927 03:18:19.154668 25677 ts_tablet_manager.cc:1905] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.154690 25676 raft_consensus.cc:2229] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.154723 25676 raft_consensus.cc:2258] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.156163 25677 ts_tablet_manager.cc:1918] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:18:19.156225 25677 log.cc:1196] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/8ac730bf107d4ee58f321482680ddc28
I20260927 03:18:19.156297 25678 ts_tablet_manager.cc:1918] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:18:19.156351 25678 log.cc:1196] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/8ac730bf107d4ee58f321482680ddc28
I20260927 03:18:19.156455 25677 ts_tablet_manager.cc:1939] T 8ac730bf107d4ee58f321482680ddc28 P 21b594dc5ebc4508a30d62017bed6a73: Deleting consensus metadata
I20260927 03:18:19.156591 25678 ts_tablet_manager.cc:1939] T 8ac730bf107d4ee58f321482680ddc28 P 45806b928b96421195bdb84b82d42bb7: Deleting consensus metadata
I20260927 03:18:19.156783 25130 catalog_manager.cc:4841] TS 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339): tablet 8ac730bf107d4ee58f321482680ddc28 (table test-table [id=a7a7dfccac5f45d89708de3371613451]) successfully deleted
I20260927 03:18:19.156927 25132 catalog_manager.cc:4841] TS 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825): tablet 8ac730bf107d4ee58f321482680ddc28 (table test-table [id=a7a7dfccac5f45d89708de3371613451]) successfully deleted
I20260927 03:18:19.158157 25676 ts_tablet_manager.cc:1905] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.159407 25676 ts_tablet_manager.cc:1918] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:18:19.159466 25676 log.cc:1196] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/8ac730bf107d4ee58f321482680ddc28
I20260927 03:18:19.159672 25676 ts_tablet_manager.cc:1939] T 8ac730bf107d4ee58f321482680ddc28 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting consensus metadata
I20260927 03:18:19.160003 25131 catalog_manager.cc:4841] TS 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437): tablet 8ac730bf107d4ee58f321482680ddc28 (table test-table [id=a7a7dfccac5f45d89708de3371613451]) successfully deleted
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:18:19.178779 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:19.173382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.17.52.190:34615)
0927 03:18:19.173451 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.173454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.173561 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:18:19.173648 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.173649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.173660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.173726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:19.174281 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.174781 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.175598 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.175795 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.175865 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.178691 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42576: BlockingRecv error: recv got EOF from 127.0.0.1:42576 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM
I20260927 03:18:19.186699 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.179573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.17.52.190:34615)
0927 03:18:19.179611 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.179613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.179714 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:18:19.179751 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.179751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.179761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.179824 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:19.180249 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.181080 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.181745 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.182039 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.182115 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.184039 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:19.184040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:19.184041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:19.184048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:19.185690 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:19.185988 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:19.185989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:19.185990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:19.186052 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:19.186222 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:19.186223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:19.186223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:19.186364 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:19.186457 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.186537 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.186608 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:19.188244 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:19.188418 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:19.189846 25562 tablet_service.cc:1467] Processing CreateTablet for tablet 18c299c1824641429dba86bf27662a64 (DEFAULT_TABLE table=test-table [id=b4bc999e439047fc920bbcf36969ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.189908 25421 tablet_service.cc:1467] Processing CreateTablet for tablet 18c299c1824641429dba86bf27662a64 (DEFAULT_TABLE table=test-table [id=b4bc999e439047fc920bbcf36969ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.189926 25282 tablet_service.cc:1467] Processing CreateTablet for tablet 18c299c1824641429dba86bf27662a64 (DEFAULT_TABLE table=test-table [id=b4bc999e439047fc920bbcf36969ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.189976 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c299c1824641429dba86bf27662a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.190012 25421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c299c1824641429dba86bf27662a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.190048 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c299c1824641429dba86bf27662a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.191286 25653 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Bootstrap starting.
I20260927 03:18:19.191366 25654 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Bootstrap starting.
I20260927 03:18:19.191566 25653 tablet_bootstrap.cc:654] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.191768 25654 tablet_bootstrap.cc:654] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.191838 25655 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Bootstrap starting.
I20260927 03:18:19.192432 25655 tablet_bootstrap.cc:654] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.192876 25653 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: No bootstrap required, opened a new log
I20260927 03:18:19.192931 25653 ts_tablet_manager.cc:1397] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:19.193035 25653 raft_consensus.cc:348] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.193104 25653 raft_consensus.cc:374] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.193123 25653 raft_consensus.cc:729] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.193152 25655 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: No bootstrap required, opened a new log
I20260927 03:18:19.193161 25653 consensus_queue.cc:260] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.193195 25655 ts_tablet_manager.cc:1397] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:19.193238 25653 ts_tablet_manager.cc:1428] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.193346 25655 raft_consensus.cc:348] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.193432 25655 raft_consensus.cc:374] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.193459 25655 raft_consensus.cc:729] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7403ea84feaa44f4b1941e5b389cdb0b, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.193502 25655 consensus_queue.cc:260] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.193615 25655 ts_tablet_manager.cc:1428] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.193686 25654 tablet_bootstrap.cc:492] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: No bootstrap required, opened a new log
I20260927 03:18:19.193733 25654 ts_tablet_manager.cc:1397] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:19.193876 25654 raft_consensus.cc:348] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.193938 25654 raft_consensus.cc:374] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.193957 25654 raft_consensus.cc:729] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.193995 25654 consensus_queue.cc:260] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.194073 25654 ts_tablet_manager.cc:1428] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.209331 25661 raft_consensus.cc:482] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:19.209425 25661 raft_consensus.cc:504] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.209735 25661 leader_election.cc:288] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.213361 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.210024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:52675 (local address 127.17.52.131:34437)
0927 03:18:19.210065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.210068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.210153 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:18:19.210252 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.210254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.210265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.210315 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:19.210447 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.210912 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.212348 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.212956 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.213037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.213045 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.213234 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.213284 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:19.213409 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.209972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:51181 (local address 127.17.52.130:35825)
0927 03:18:19.210025 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.210028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.210128 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:18:19.210199 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.210200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.210210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.210268 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:19.210404 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.211136 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.212636 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.213038 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.213134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.213144 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.213302 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.213344 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:18:19.213631 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.209896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35825 (local address 127.17.52.129:51181)
0927 03:18:19.210069 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.210079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.210139 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.210268 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.210269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.210270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:19.210344 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.210346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.211220 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.211221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.212522 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.212523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.213137 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.213138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.213216 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.213220 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.213512 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.213551 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260927 03:18:19.213944 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.209901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34437 (local address 127.17.52.129:52675)
0927 03:18:19.210093 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.210102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.210159 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.210325 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.210325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.210327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:19.210390 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.210392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.210990 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.210991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.212246 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.212249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.213063 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.213065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.213140 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.213144 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.213798 (+ 654us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.213845 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260927 03:18:19.213886 25442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c299c1824641429dba86bf27662a64" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45806b928b96421195bdb84b82d42bb7" is_pre_election: true
I20260927 03:18:19.214054 25442 raft_consensus.cc:2454] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 0.
I20260927 03:18:19.214205 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c299c1824641429dba86bf27662a64" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" is_pre_election: true
I20260927 03:18:19.214272 25236 leader_election.cc:302] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:19.214310 25581 raft_consensus.cc:2454] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 0.
I20260927 03:18:19.214395 25661 raft_consensus.cc:2781] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:19.214434 25661 raft_consensus.cc:482] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:19.214457 25661 raft_consensus.cc:3037] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.215131 25661 raft_consensus.cc:504] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.215266 25661 leader_election.cc:288] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 election: Requested vote from peers 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.215425 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c299c1824641429dba86bf27662a64" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
I20260927 03:18:19.215443 25442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c299c1824641429dba86bf27662a64" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45806b928b96421195bdb84b82d42bb7"
I20260927 03:18:19.215513 25442 raft_consensus.cc:3037] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.215513 25581 raft_consensus.cc:3037] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.216125 25581 raft_consensus.cc:2454] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 1.
I20260927 03:18:19.216292 25235 leader_election.cc:302] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 7403ea84feaa44f4b1941e5b389cdb0b; no voters:
I20260927 03:18:19.216312 25442 raft_consensus.cc:2454] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 1.
I20260927 03:18:19.216372 25661 raft_consensus.cc:2781] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:19.216543 25661 raft_consensus.cc:686] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 LEADER]: Becoming Leader. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Running, Role: LEADER
I20260927 03:18:19.216637 25661 consensus_queue.cc:237] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.217309 25144 catalog_manager.cc:5495] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "21b594dc5ebc4508a30d62017bed6a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:19.223205 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.220919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.17.52.129:36339)
0927 03:18:19.220964 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.220966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.220976 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:19.221020 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.221020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.221037 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.221115 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:19.221212 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.221663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.222435 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.222649 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.222735 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.222886 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.222996 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.223077 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.223118 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.223135 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:19.223997 25581 raft_consensus.cc:1261] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Refusing update from remote peer 21b594dc5ebc4508a30d62017bed6a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.224131 25442 raft_consensus.cc:1261] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Refusing update from remote peer 21b594dc5ebc4508a30d62017bed6a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.224423 25720 consensus_queue.cc:1035] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.224537 25661 consensus_queue.cc:1035] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.227733 25144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.227831 25144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.228389 25144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Sending DeleteTablet for 3 replicas of tablet 18c299c1824641429dba86bf27662a64
I20260927 03:18:19.228674 25282 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c299c1824641429dba86bf27662a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:35196
I20260927 03:18:19.228672 25562 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c299c1824641429dba86bf27662a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260927 03:18:19.228751 25677 tablet_replica.cc:330] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: stopping tablet replica
I20260927 03:18:19.228751 25676 tablet_replica.cc:330] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: stopping tablet replica
I20260927 03:18:19.228797 25677 raft_consensus.cc:2229] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:19.228806 25676 raft_consensus.cc:2229] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.228845 25676 raft_consensus.cc:2258] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.228878 25677 raft_consensus.cc:2258] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.229158 25677 ts_tablet_manager.cc:1905] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.229425 25676 ts_tablet_manager.cc:1905] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.230710 25677 ts_tablet_manager.cc:1918] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:19.230762 25677 log.cc:1196] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/18c299c1824641429dba86bf27662a64
I20260927 03:18:19.230825 25676 ts_tablet_manager.cc:1918] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:19.230873 25676 log.cc:1196] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/18c299c1824641429dba86bf27662a64
I20260927 03:18:19.230983 25677 ts_tablet_manager.cc:1939] T 18c299c1824641429dba86bf27662a64 P 21b594dc5ebc4508a30d62017bed6a73: Deleting consensus metadata
I20260927 03:18:19.231073 25676 ts_tablet_manager.cc:1939] T 18c299c1824641429dba86bf27662a64 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting consensus metadata
I20260927 03:18:19.231194 25421 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c299c1824641429dba86bf27662a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260927 03:18:19.231292 25678 tablet_replica.cc:330] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: stopping tablet replica
I20260927 03:18:19.231339 25678 raft_consensus.cc:2229] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.231375 25678 raft_consensus.cc:2258] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.231559 25131 catalog_manager.cc:4841] TS 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437): tablet 18c299c1824641429dba86bf27662a64 (table test-table [id=b4bc999e439047fc920bbcf36969ae05]) successfully deleted
I20260927 03:18:19.231762 25130 catalog_manager.cc:4841] TS 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339): tablet 18c299c1824641429dba86bf27662a64 (table test-table [id=b4bc999e439047fc920bbcf36969ae05]) successfully deleted
I20260927 03:18:19.231971 25678 ts_tablet_manager.cc:1905] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.233075 25678 ts_tablet_manager.cc:1918] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:19.233143 25678 log.cc:1196] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/18c299c1824641429dba86bf27662a64
I20260927 03:18:19.233335 25678 ts_tablet_manager.cc:1939] T 18c299c1824641429dba86bf27662a64 P 45806b928b96421195bdb84b82d42bb7: Deleting consensus metadata
I20260927 03:18:19.233646 25132 catalog_manager.cc:4841] TS 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825): tablet 18c299c1824641429dba86bf27662a64 (table test-table [id=b4bc999e439047fc920bbcf36969ae05]) successfully deleted
I20260927 03:18:19.233611 25144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:19.233775 25144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:19.234907 25282 tablet_service.cc:1467] Processing CreateTablet for tablet e2d22432206c4b7fb1160bd409653e60 (DEFAULT_TABLE table=test-table [id=b7c08e3423e44bcdb73add3a1bec2a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.234911 25421 tablet_service.cc:1467] Processing CreateTablet for tablet e2d22432206c4b7fb1160bd409653e60 (DEFAULT_TABLE table=test-table [id=b7c08e3423e44bcdb73add3a1bec2a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.235020 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d22432206c4b7fb1160bd409653e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.235020 25421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d22432206c4b7fb1160bd409653e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.235975 25653 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Bootstrap starting.
I20260927 03:18:19.236063 25562 tablet_service.cc:1467] Processing CreateTablet for tablet e2d22432206c4b7fb1160bd409653e60 (DEFAULT_TABLE table=test-table [id=b7c08e3423e44bcdb73add3a1bec2a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:19.236157 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d22432206c4b7fb1160bd409653e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:19.236213 25654 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: Bootstrap starting.
I20260927 03:18:19.236270 25653 tablet_bootstrap.cc:654] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.236624 25654 tablet_bootstrap.cc:654] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.237347 25654 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: No bootstrap required, opened a new log
I20260927 03:18:19.237372 25655 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Bootstrap starting.
I20260927 03:18:19.237394 25654 ts_tablet_manager.cc:1397] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:18:19.237536 25654 raft_consensus.cc:348] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.237591 25654 raft_consensus.cc:374] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.237609 25654 raft_consensus.cc:729] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.237650 25654 consensus_queue.cc:260] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.237730 25654 ts_tablet_manager.cc:1428] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.237766 25655 tablet_bootstrap.cc:654] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:19.237799 25653 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: No bootstrap required, opened a new log
I20260927 03:18:19.237841 25653 ts_tablet_manager.cc:1397] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:19.237946 25653 raft_consensus.cc:348] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.238011 25653 raft_consensus.cc:374] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.238037 25653 raft_consensus.cc:729] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45806b928b96421195bdb84b82d42bb7, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.238081 25653 consensus_queue.cc:260] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.238164 25653 ts_tablet_manager.cc:1428] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.238358 25655 tablet_bootstrap.cc:492] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: No bootstrap required, opened a new log
I20260927 03:18:19.238404 25655 ts_tablet_manager.cc:1397] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:19.238541 25655 raft_consensus.cc:348] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.238605 25655 raft_consensus.cc:374] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:19.238627 25655 raft_consensus.cc:729] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7403ea84feaa44f4b1941e5b389cdb0b, State: Initialized, Role: FOLLOWER
I20260927 03:18:19.238668 25655 consensus_queue.cc:260] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.238754 25655 ts_tablet_manager.cc:1428] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:19.362442 25661 raft_consensus.cc:482] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:19.362558 25661 raft_consensus.cc:504] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.362789 25661 leader_election.cc:288] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.362959 25442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d22432206c4b7fb1160bd409653e60" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45806b928b96421195bdb84b82d42bb7" is_pre_election: true
I20260927 03:18:19.363037 25442 raft_consensus.cc:2454] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 0.
I20260927 03:18:19.363050 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d22432206c4b7fb1160bd409653e60" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" is_pre_election: true
I20260927 03:18:19.363183 25581 raft_consensus.cc:2454] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 0.
I20260927 03:18:19.363227 25236 leader_election.cc:302] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 45806b928b96421195bdb84b82d42bb7; no voters:
I20260927 03:18:19.363340 25661 raft_consensus.cc:2781] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:19.363375 25661 raft_consensus.cc:482] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:19.363406 25661 raft_consensus.cc:3037] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.364043 25661 raft_consensus.cc:504] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.364182 25661 leader_election.cc:288] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 election: Requested vote from peers 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825), 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437)
I20260927 03:18:19.364326 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d22432206c4b7fb1160bd409653e60" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7403ea84feaa44f4b1941e5b389cdb0b"
I20260927 03:18:19.364355 25442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d22432206c4b7fb1160bd409653e60" candidate_uuid: "21b594dc5ebc4508a30d62017bed6a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45806b928b96421195bdb84b82d42bb7"
I20260927 03:18:19.364396 25581 raft_consensus.cc:3037] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.364442 25442 raft_consensus.cc:3037] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:19.365067 25581 raft_consensus.cc:2454] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 1.
I20260927 03:18:19.365085 25442 raft_consensus.cc:2454] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b594dc5ebc4508a30d62017bed6a73 in term 1.
I20260927 03:18:19.365239 25235 leader_election.cc:302] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b594dc5ebc4508a30d62017bed6a73, 7403ea84feaa44f4b1941e5b389cdb0b; no voters:
I20260927 03:18:19.365334 25661 raft_consensus.cc:2781] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:19.365397 25661 raft_consensus.cc:686] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 LEADER]: Becoming Leader. State: Replica: 21b594dc5ebc4508a30d62017bed6a73, State: Running, Role: LEADER
I20260927 03:18:19.365442 25661 consensus_queue.cc:237] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } }
I20260927 03:18:19.365971 25144 catalog_manager.cc:5495] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "21b594dc5ebc4508a30d62017bed6a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b594dc5ebc4508a30d62017bed6a73" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:19.387132 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.385105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:37959 (local address 127.17.52.130:35825)
0927 03:18:19.385152 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.385154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.385289 (+ 135us) server_negotiation.cc:407] Connection header received
0927 03:18:19.385329 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.385329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.385339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.385386 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:19.385469 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.385926 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.386635 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.386805 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.386852 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.386975 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.386998 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.387031 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.387064 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.387077 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:18:19.387158 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.385049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35825 (local address 127.17.52.129:37959)
0927 03:18:19.385224 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.385233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.385288 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.385388 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.385389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.385390 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:18:19.385433 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.385435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.385985 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.385985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.386573 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.386574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.386866 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.386867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.386906 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.386918 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.387032 (+ 114us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.387033 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.387063 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.387100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260927 03:18:19.389240 25442 raft_consensus.cc:1261] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Refusing update from remote peer 21b594dc5ebc4508a30d62017bed6a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.389264 25581 raft_consensus.cc:1261] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Refusing update from remote peer 21b594dc5ebc4508a30d62017bed6a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:19.389443 25661 consensus_queue.cc:1035] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45806b928b96421195bdb84b82d42bb7" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.389536 25720 consensus_queue.cc:1035] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7403ea84feaa44f4b1941e5b389cdb0b" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:19.395762 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.393539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41937 (local address 127.17.52.129:36339)
0927 03:18:19.393581 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:19.393583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:19.393627 (+ 44us) server_negotiation.cc:407] Connection header received
0927 03:18:19.393710 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:19.393711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:19.393721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:19.393774 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:19.393887 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.394376 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.395085 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.395306 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.395377 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.395500 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.395537 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.395591 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:19.395664 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:18:19.395686 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:19.395797 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.393512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36339 (local address 127.17.52.130:41937)
0927 03:18:19.393583 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.393593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:19.393642 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:19.393787 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:19.393788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:19.393789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:18:19.393831 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.393832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.394454 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.394455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.395006 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:19.395007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:19.395375 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:19.395376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:19.395433 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:19.395440 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:18:19.395604 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:18:19.395605 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:18:19.395658 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:18:19.395706 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 03:18:19.401633 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.401718 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:19.402288 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 595f0f86f5a74d829dc86d9ab29582d7: Sending DeleteTablet for 3 replicas of tablet e2d22432206c4b7fb1160bd409653e60
I20260927 03:18:19.402496 25282 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d22432206c4b7fb1160bd409653e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:35196
I20260927 03:18:19.402532 25421 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d22432206c4b7fb1160bd409653e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260927 03:18:19.402592 25562 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d22432206c4b7fb1160bd409653e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:19 UTC) from {username='oryx', principal='oryx/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260927 03:18:19.402688 25677 tablet_replica.cc:330] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73: stopping tablet replica
I20260927 03:18:19.402750 25677 raft_consensus.cc:2229] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:19.402777 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25217
I20260927 03:18:19.402822 25677 raft_consensus.cc:2258] T e2d22432206c4b7fb1160bd409653e60 P 21b594dc5ebc4508a30d62017bed6a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.402895 25678 tablet_replica.cc:330] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: stopping tablet replica
I20260927 03:18:19.402952 25678 raft_consensus.cc:2229] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.402984 25678 raft_consensus.cc:2258] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.403045 25676 tablet_replica.cc:330] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: stopping tablet replica
I20260927 03:18:19.403095 25676 raft_consensus.cc:2229] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:19.403126 25676 raft_consensus.cc:2258] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:19.404886 25678 ts_tablet_manager.cc:1905] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.406023 25676 ts_tablet_manager.cc:1905] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:19.406193 25678 ts_tablet_manager.cc:1918] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:18:19.406239 25678 log.cc:1196] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/e2d22432206c4b7fb1160bd409653e60
I20260927 03:18:19.406450 25678 ts_tablet_manager.cc:1939] T e2d22432206c4b7fb1160bd409653e60 P 45806b928b96421195bdb84b82d42bb7: Deleting consensus metadata
I20260927 03:18:19.406733 25132 catalog_manager.cc:4841] TS 45806b928b96421195bdb84b82d42bb7 (127.17.52.130:35825): tablet e2d22432206c4b7fb1160bd409653e60 (table test-table [id=b7c08e3423e44bcdb73add3a1bec2a4f]) successfully deleted
I20260927 03:18:19.407234 25676 ts_tablet_manager.cc:1918] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:18:19.407307 25676 log.cc:1196] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/e2d22432206c4b7fb1160bd409653e60
I20260927 03:18:19.407500 25676 ts_tablet_manager.cc:1939] T e2d22432206c4b7fb1160bd409653e60 P 7403ea84feaa44f4b1941e5b389cdb0b: Deleting consensus metadata
I20260927 03:18:19.407780 25131 catalog_manager.cc:4841] TS 7403ea84feaa44f4b1941e5b389cdb0b (127.17.52.131:34437): tablet e2d22432206c4b7fb1160bd409653e60 (table test-table [id=b7c08e3423e44bcdb73add3a1bec2a4f]) successfully deleted
W20260927 03:18:19.412660 25130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:36339 (error 108)
I20260927 03:18:19.412899 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25357
I20260927 03:18:19.413565 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:19.413003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36339 (local address 127.0.0.1:35220)
0927 03:18:19.413468 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:19.413520 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":37,"threads_started":1}
W20260927 03:18:19.414184 25130 catalog_manager.cc:4584] TS 21b594dc5ebc4508a30d62017bed6a73 (127.17.52.129:36339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2d22432206c4b7fb1160bd409653e60: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36339: connect: Connection refused (error 111)
I20260927 03:18:19.418610 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25497
I20260927 03:18:19.423425 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25113
2026-09-27T03:18:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2436 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25740](info): set up 2 sockets
Sep 27 03:18:19 dist-test-slave-7j43 krb5kdc[25740](info): commencing operation
krb5kdc: starting...
W20260927 03:18:20.457545 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:20 dist-test-slave-7j43 krb5kdc[25740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:20.569537 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38351
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:20.649456 25753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:20.649700 25753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:20.649737 25753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:20.650880 25753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:20.650944 25753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:20.650974 25753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:20.651002 25753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:20.651031 25753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:20.652441 25753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:20.652732 25753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:20.652951 25753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:20.657639 25761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:20.657696 25759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:20.657850 25758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:20.657820 25753 server_base.cc:1034] running on GCE node
I20260927 03:18:20.658100 25753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:20.658345 25753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:20.659488 25753 hybrid_clock.cc:648] HybridClock initialized: now 1790479100659474 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T03:18:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1242 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:20 dist-test-slave-7j43 krb5kdc[25766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:20 dist-test-slave-7j43 krb5kdc[25766](info): set up 2 sockets
Sep 27 03:18:20 dist-test-slave-7j43 krb5kdc[25766](info): commencing operation
krb5kdc: starting...
W20260927 03:18:21.741611 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:21 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:21.848887 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45347
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:21.927802 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:21.928047 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:21.928085 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:21.929229 25779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:21.929302 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:21.929344 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:21.929385 25779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:21.929414 25779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:21.930872 25779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:45347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:21.931134 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:21.931355 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:21.935949 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:21.935945 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:21.936450 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:21.936527 25779 server_base.cc:1034] running on GCE node
I20260927 03:18:21.936771 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:21.937047 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:21.938201 25779 hybrid_clock.cc:648] HybridClock initialized: now 1790479101938189 us; error 28 us; skew 500 ppm
Sep 27 03:18:21 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479101, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:21.940668 25779 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:21.941211 25779 webserver.cc:460] Webserver started at http://127.17.52.190:35223/ using document root <none> and password file <none>
I20260927 03:18:21.941486 25779 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:21.941547 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:21.941679 25779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:21.942690 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "a157a50840ce4969840d3c08cfc9173b"
format_stamp: "Formatted at 2026-09-27 03:18:21 on dist-test-slave-7j43"
server_key: "00c2c05c914899ff0f3ff22a553fbc66"
server_key_iv: "1c7a986fd6d5ef5564d0e7dd7d824cfa"
server_key_version: "encryptionkey@0"
I20260927 03:18:21.943037 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "a157a50840ce4969840d3c08cfc9173b"
format_stamp: "Formatted at 2026-09-27 03:18:21 on dist-test-slave-7j43"
server_key: "00c2c05c914899ff0f3ff22a553fbc66"
server_key_iv: "1c7a986fd6d5ef5564d0e7dd7d824cfa"
server_key_version: "encryptionkey@0"
I20260927 03:18:21.944304 25779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:21.945052 25794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:21.945220 25779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:21.945329 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "a157a50840ce4969840d3c08cfc9173b"
format_stamp: "Formatted at 2026-09-27 03:18:21 on dist-test-slave-7j43"
server_key: "00c2c05c914899ff0f3ff22a553fbc66"
server_key_iv: "1c7a986fd6d5ef5564d0e7dd7d824cfa"
server_key_version: "encryptionkey@0"
I20260927 03:18:21.945407 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:21.958242 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:21.960461 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:21.960695 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:21.964562 25779 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:45347
I20260927 03:18:21.964651 25846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:45347 every 8 connection(s)
I20260927 03:18:21.965027 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:21.965924 25847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:21.968163 25847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Bootstrap starting.
I20260927 03:18:21.968739 25847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:21.968971 25847 log.cc:824] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:21.969630 25847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: No bootstrap required, opened a new log
I20260927 03:18:21.971151 25847 raft_consensus.cc:348] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:21.971285 25847 raft_consensus.cc:374] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:21.971307 25847 raft_consensus.cc:729] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157a50840ce4969840d3c08cfc9173b, State: Initialized, Role: FOLLOWER
I20260927 03:18:21.971446 25847 consensus_queue.cc:260] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:21.971522 25847 raft_consensus.cc:388] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:21.971544 25847 raft_consensus.cc:482] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:21.971580 25847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:21.972045 25847 raft_consensus.cc:504] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:21.972133 25847 leader_election.cc:302] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a157a50840ce4969840d3c08cfc9173b; no voters:
I20260927 03:18:21.972259 25847 leader_election.cc:288] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:21.972322 25850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:21.972508 25850 raft_consensus.cc:686] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 LEADER]: Becoming Leader. State: Replica: a157a50840ce4969840d3c08cfc9173b, State: Running, Role: LEADER
I20260927 03:18:21.972496 25847 sys_catalog.cc:564] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:21.972687 25850 consensus_queue.cc:237] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:21.973028 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25779
I20260927 03:18:21.973042 25852 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a157a50840ce4969840d3c08cfc9173b. Latest consensus state: current_term: 1 leader_uuid: "a157a50840ce4969840d3c08cfc9173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } } }
I20260927 03:18:21.973116 25852 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:21.973143 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:21.973165 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a157a50840ce4969840d3c08cfc9173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } } }
I20260927 03:18:21.973223 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:21.973361 17618 external_mini_cluster.cc:1428] Setting key 2ae8ea76bb62b3d52515d8007f15964c
I20260927 03:18:21.973799 25856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:21.974318 25856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:21.975883 25856 catalog_manager.cc:1347] Generated new cluster ID: 2ed231395779439680fc43f0866d259f
I20260927 03:18:21.975936 25856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:18:21 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:21.983700 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:21.974187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.17.52.190:45347)
0927 03:18:21.974574 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:21.974584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:21.974618 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:18:21.974885 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:21.974891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:21.974958 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:21.975088 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:21.975946 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:21.976997 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:21.977745 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:21.978168 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:21.978290 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:21.980472 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:21.980476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:21.980477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:21.980489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:21.982142 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:21.982502 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:21.982505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:21.982506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:21.982606 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:21.982820 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:21.982821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:21.982821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:21.983054 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:21.983216 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:21.983315 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:18:21.983546 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":62,"threads_started":1}
I20260927 03:18:21.984627 25856 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:21.985100 25856 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:21.992031 25856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Generated new TSK 0
I20260927 03:18:21.992177 25856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:22.020419 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:22.099298 25873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.099488 25873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.099511 25873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.100646 25873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.100683 25873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.100729 25873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:22.102088 25873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.102319 25873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.102530 25873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.107869 25878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.107858 25881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.107949 25879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.107988 25873 server_base.cc:1034] running on GCE node
I20260927 03:18:22.108283 25873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.108525 25873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.109702 25873 hybrid_clock.cc:648] HybridClock initialized: now 1790479102109693 us; error 29 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.112095 25873 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:22.112499 25873 webserver.cc:460] Webserver started at http://127.17.52.129:41621/ using document root <none> and password file <none>
I20260927 03:18:22.112718 25873 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.112762 25873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.112859 25873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:22.113761 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "957985af32e148efade5b3ed534e8b26"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "cd902fd9f446365eb322f7f9354a7ebd"
server_key_iv: "000434d4afd1372d9efae69cb9df97e8"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.114048 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "957985af32e148efade5b3ed534e8b26"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "cd902fd9f446365eb322f7f9354a7ebd"
server_key_iv: "000434d4afd1372d9efae69cb9df97e8"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.115314 25873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:22.115906 25888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.116055 25873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:22.116142 25873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "957985af32e148efade5b3ed534e8b26"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "cd902fd9f446365eb322f7f9354a7ebd"
server_key_iv: "000434d4afd1372d9efae69cb9df97e8"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.116210 25873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.149673 25873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.151841 25873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.151993 25873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.152217 25873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:22.152518 25873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:22.152565 25873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.152650 25873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:22.152683 25873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.158324 25873 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:42883
I20260927 03:18:22.158371 26001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:42883 every 8 connection(s)
I20260927 03:18:22.158701 25873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:22.164966 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 25873
I20260927 03:18:22.165081 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:22.165287 17618 external_mini_cluster.cc:1428] Setting key e7ba05f3de6c1c749908ddd31f605497
I20260927 03:18:22.168012 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.159323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:41017 (local address 127.17.52.190:45347)
0927 03:18:22.159364 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.159365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.159577 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:18:22.160112 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.160113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.160130 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.160183 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.161180 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.161699 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.162682 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.162895 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.162962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.165223 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:22.165224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:22.165225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:22.165232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:22.166674 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.167072 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.167073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.167074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.167140 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.167372 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.167373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.167373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.167485 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:22.167556 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:22.167827 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 03:18:22.167923 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:18:22.168228 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.159300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:41017)
0927 03:18:22.159459 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.159494 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.159989 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.160258 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.160263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.160882 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.161070 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.161074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.161790 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.161791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.162586 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.162589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.162980 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.162981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.163108 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.163554 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.163562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.165145 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:22.166767 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.166773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.166775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.166986 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.167201 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.167202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.167202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.167308 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.167560 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:22.167566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:22.167704 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:18:22.167805 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:18:22.167953 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260927 03:18:22.168699 26002 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45347
I20260927 03:18:22.168800 26002 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:22.168962 26002 heartbeater.cc:511] Master 127.17.52.190:45347 requested a full tablet report, sending...
I20260927 03:18:22.169549 25811 ts_manager.cc:194] Registered new tserver with Master: 957985af32e148efade5b3ed534e8b26 (127.17.52.129:42883)
I20260927 03:18:22.170498 25811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:41017
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:22.188048 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:22.267818 26007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.268045 26007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.268082 26007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.269309 26007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.269379 26007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.269428 26007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:22.270864 26007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.271121 26007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.271332 26007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.276320 26013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.276283 26012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.276283 26015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.276561 26007 server_base.cc:1034] running on GCE node
I20260927 03:18:22.276809 26007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.277033 26007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.278193 26007 hybrid_clock.cc:648] HybridClock initialized: now 1790479102278173 us; error 40 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.280609 26007 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:22.281023 26007 webserver.cc:460] Webserver started at http://127.17.52.130:40063/ using document root <none> and password file <none>
I20260927 03:18:22.281247 26007 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.281312 26007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.281440 26007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:22.282449 26007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "8102e87d5ce449e98de6b7e00ddcb8e9"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "1f7cbed605cb29fd36b97cc54c1c984e"
server_key_iv: "36e0ea6d2e0ef8779c61cd9887d31bc2"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.282804 26007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "8102e87d5ce449e98de6b7e00ddcb8e9"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "1f7cbed605cb29fd36b97cc54c1c984e"
server_key_iv: "36e0ea6d2e0ef8779c61cd9887d31bc2"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.284047 26007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:22.284742 26022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.284917 26007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:22.285007 26007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "8102e87d5ce449e98de6b7e00ddcb8e9"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "1f7cbed605cb29fd36b97cc54c1c984e"
server_key_iv: "36e0ea6d2e0ef8779c61cd9887d31bc2"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.285085 26007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.297233 26007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.298939 26007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.299186 26007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.299392 26007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:22.299679 26007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:22.299729 26007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.299769 26007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:22.299799 26007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.305459 26007 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:36719
I20260927 03:18:22.305493 26135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:36719 every 8 connection(s)
I20260927 03:18:22.305814 26007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:22.312178 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26007
I20260927 03:18:22.312306 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:22.312530 17618 external_mini_cluster.cc:1428] Setting key 355694fc2fe103d71c9356ef6636b264
I20260927 03:18:22.314993 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.306495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:46917 (local address 127.17.52.190:45347)
0927 03:18:22.306542 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.306544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.306744 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:18:22.307181 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.307182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.307192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.307245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.308124 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.308695 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.309655 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.309845 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.309903 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.311818 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:22.311818 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:22.311819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:22.311825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:22.313370 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.313831 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.313832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.313832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.313880 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.314245 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.314246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.314246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.314370 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:22.314433 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:22.314779 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 03:18:22.314873 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:22.315028 26138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.306478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:46917)
0927 03:18:22.306631 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.306664 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.307102 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.307310 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.307313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.307850 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.308033 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.308037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.308784 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.308785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.309572 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.309573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.309914 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.309915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.310007 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.310492 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.310500 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.311741 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:22.313438 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.313444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.313446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.313757 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.313939 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.313940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.313941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.314075 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.314439 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:22.314445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:22.314603 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 03:18:22.314727 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:18:22.314878 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260927 03:18:22.315547 26136 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45347
I20260927 03:18:22.315632 26136 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:22.315783 26136 heartbeater.cc:511] Master 127.17.52.190:45347 requested a full tablet report, sending...
I20260927 03:18:22.316119 25811 ts_manager.cc:194] Registered new tserver with Master: 8102e87d5ce449e98de6b7e00ddcb8e9 (127.17.52.130:36719)
I20260927 03:18:22.316473 25811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:46917
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:22.335300 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:22.413903 26141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.414093 26141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.414115 26141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.415179 26141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.415213 26141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.415238 26141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:22.416592 26141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.416779 26141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.416937 26141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.421452 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.421500 26141 server_base.cc:1034] running on GCE node
W20260927 03:18:22.421483 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.421475 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.421794 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.422017 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.423166 26141 hybrid_clock.cc:648] HybridClock initialized: now 1790479102423151 us; error 32 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.425437 26141 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:22.425868 26141 webserver.cc:460] Webserver started at http://127.17.52.131:39035/ using document root <none> and password file <none>
I20260927 03:18:22.426076 26141 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.426136 26141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.426278 26141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:22.427228 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "ac68285d8f3f454c9e1b12e844bd60cd"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "69e74b7a28a3cc55fa9203e2652ae30e"
server_key_iv: "9527a2df362890ed343f34b29279574b"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.427577 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "ac68285d8f3f454c9e1b12e844bd60cd"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "69e74b7a28a3cc55fa9203e2652ae30e"
server_key_iv: "9527a2df362890ed343f34b29279574b"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.428850 26141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:22.429540 26156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.429692 26141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:22.429787 26141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "ac68285d8f3f454c9e1b12e844bd60cd"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "69e74b7a28a3cc55fa9203e2652ae30e"
server_key_iv: "9527a2df362890ed343f34b29279574b"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.429852 26141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.442653 26141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.444324 26141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.444473 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.444725 26141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:22.445001 26141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:22.445055 26141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.445096 26141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:22.445122 26141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.450860 26141 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:36159
I20260927 03:18:22.450906 26269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:36159 every 8 connection(s)
I20260927 03:18:22.451188 26141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:22.459524 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.451792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:38983 (local address 127.17.52.190:45347)
0927 03:18:22.451849 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.451851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.452045 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:18:22.452534 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.452535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.452549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.452629 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.453504 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.454011 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.454988 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.455196 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.455277 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.457120 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:22.457122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:22.457122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:22.457130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:22.458256 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.458629 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.458630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.458631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.458688 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:22.458922 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:22.458922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:22.458923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:22.459031 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:22.459096 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:22.459356 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:18:22.459434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:18:22.459653 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26141
I20260927 03:18:22.459712 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.451776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:38983)
0927 03:18:22.451937 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.451969 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.452435 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.452697 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.452701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.453228 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.453408 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.453412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.454115 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.454116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.454901 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.454903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.455290 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.455291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.455410 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.455791 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.455799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.457027 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:22.458347 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.458353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.458355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.458559 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.458749 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:22.458750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:22.458750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:22.458853 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:22.459100 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:22.459106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:22.459229 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 03:18:22.459335 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:18:22.459480 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260927 03:18:22.459770 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:22.459998 17618 external_mini_cluster.cc:1428] Setting key 43cd61500289e67fd0b829c84f00c924
I20260927 03:18:22.460182 26270 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45347
I20260927 03:18:22.460299 26270 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:22.460486 26270 heartbeater.cc:511] Master 127.17.52.190:45347 requested a full tablet report, sending...
I20260927 03:18:22.460904 25811 ts_manager.cc:194] Registered new tserver with Master: ac68285d8f3f454c9e1b12e844bd60cd (127.17.52.131:36159)
I20260927 03:18:22.461262 25811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:38983
I20260927 03:18:22.471490 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
Entry for principal oryx/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab.
WARNING: no policy specified for oryx/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
Entry for principal oryx/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab.
WARNING: no policy specified for oryx/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
Entry for principal oryx/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab.
I20260927 03:18:22.533212 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25873
I20260927 03:18:22.537433 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26007
I20260927 03:18:22.541419 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26141
I20260927 03:18:22.545106 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 25779
I20260927 03:18:22.550165 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45347
--webserver_interface=127.17.52.190
--webserver_port=35223
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:22.629081 26279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.629326 26279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.629367 26279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.630491 26279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:22.630554 26279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.630578 26279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.630609 26279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:22.630645 26279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:22.632042 26279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:45347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.632300 26279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.632511 26279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.637270 26287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.637292 26284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.637266 26285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.637512 26279 server_base.cc:1034] running on GCE node
I20260927 03:18:22.637768 26279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.638056 26279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.639212 26279 hybrid_clock.cc:648] HybridClock initialized: now 1790479102639211 us; error 34 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.641695 26279 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:22.642205 26279 webserver.cc:460] Webserver started at http://127.17.52.190:35223/ using document root <none> and password file <none>
I20260927 03:18:22.642454 26279 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.642525 26279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.643839 26279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:22.644522 26294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.644747 26279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:22.644843 26279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "a157a50840ce4969840d3c08cfc9173b"
format_stamp: "Formatted at 2026-09-27 03:18:21 on dist-test-slave-7j43"
server_key: "00c2c05c914899ff0f3ff22a553fbc66"
server_key_iv: "1c7a986fd6d5ef5564d0e7dd7d824cfa"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.645174 26279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.658849 26279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.660750 26279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.660897 26279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.664803 26279 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:45347
I20260927 03:18:22.664845 26346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:45347 every 8 connection(s)
I20260927 03:18:22.665227 26279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:22.666590 26347 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:18:22.667163 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Bootstrap starting.
I20260927 03:18:22.674476 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26279
I20260927 03:18:22.674854 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:42883
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:45347
--webserver_port=41621
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:18:22.675545 26347 log.cc:824] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:22.677273 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:18:22.677546 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Bootstrap complete.
I20260927 03:18:22.679733 26347 raft_consensus.cc:348] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:22.679946 26347 raft_consensus.cc:729] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157a50840ce4969840d3c08cfc9173b, State: Initialized, Role: FOLLOWER
I20260927 03:18:22.680114 26347 consensus_queue.cc:260] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:22.680184 26347 raft_consensus.cc:388] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:22.680227 26347 raft_consensus.cc:482] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:22.680280 26347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:18:22.681308 26347 raft_consensus.cc:504] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:22.681391 26347 leader_election.cc:302] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a157a50840ce4969840d3c08cfc9173b; no voters:
I20260927 03:18:22.681577 26347 leader_election.cc:288] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:18:22.681612 26351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:18:22.681784 26351 raft_consensus.cc:686] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [term 2 LEADER]: Becoming Leader. State: Replica: a157a50840ce4969840d3c08cfc9173b, State: Running, Role: LEADER
I20260927 03:18:22.681866 26347 sys_catalog.cc:564] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:22.681931 26351 consensus_queue.cc:237] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } }
I20260927 03:18:22.682219 26351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a157a50840ce4969840d3c08cfc9173b. Latest consensus state: current_term: 2 leader_uuid: "a157a50840ce4969840d3c08cfc9173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } } }
I20260927 03:18:22.682336 26351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:22.682225 26353 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a157a50840ce4969840d3c08cfc9173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157a50840ce4969840d3c08cfc9173b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45347 } } }
I20260927 03:18:22.682498 26353 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:22.683269 26360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:22.683808 26360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:22.683979 26360 catalog_manager.cc:1259] Loaded cluster ID: 2ed231395779439680fc43f0866d259f
I20260927 03:18:22.684049 26360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:22.685251 26360 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:22.685468 26360 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a157a50840ce4969840d3c08cfc9173b: Loaded TSK: 0
I20260927 03:18:22.685732 26360 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:18:22.757205 26350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.757400 26350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.757423 26350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.758605 26350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.758646 26350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.758671 26350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:22.760080 26350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.129.keytab
--principal=oryx/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:42883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=41621
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.760326 26350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.760556 26350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.766232 26373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.766237 26376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.766242 26374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.766587 26350 server_base.cc:1034] running on GCE node
I20260927 03:18:22.766780 26350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.766954 26350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.768108 26350 hybrid_clock.cc:648] HybridClock initialized: now 1790479102768079 us; error 39 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.770533 26350 init.cc:377] Logged in from keytab as oryx/127.17.52.129@KRBTEST.COM (short username oryx)
I20260927 03:18:22.770990 26350 webserver.cc:460] Webserver started at http://127.17.52.129:41621/ using document root <none> and password file <none>
I20260927 03:18:22.771219 26350 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.771275 26350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.772667 26350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:22.773259 26383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.773401 26350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:22.773481 26350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "957985af32e148efade5b3ed534e8b26"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "cd902fd9f446365eb322f7f9354a7ebd"
server_key_iv: "000434d4afd1372d9efae69cb9df97e8"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.773805 26350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.784783 26350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.786993 26350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.787125 26350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.787343 26350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:22.787667 26350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:22.787722 26350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.787755 26350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:22.787799 26350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.794881 26350 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:42883
I20260927 03:18:22.794956 26496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:42883 every 8 connection(s)
I20260927 03:18:22.795269 26350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:22.799943 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26350
I20260927 03:18:22.800237 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:36719
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:45347
--webserver_port=40063
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:22.804962 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.795887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:36445)
0927 03:18:22.796068 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.796112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.796797 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.797216 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.797221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.798051 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.798312 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.798319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.799387 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.799388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.800632 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.800635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.801447 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.801448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.801575 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.802126 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.802136 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.804781 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
W20260927 03:18:22.805143 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.795991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:36445 (local address 127.17.52.190:45347)
0927 03:18:22.796686 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.796694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.796724 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:18:22.796983 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.796988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.797026 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.797130 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.798421 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.799299 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.800706 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.801355 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.801467 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.804934 (+ 3467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:36445: BlockingRecv error: recv got EOF from 127.17.52.129:36445 (error 108)
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":72,"threads_started":1}
W20260927 03:18:22.805552 26497 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:45347 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.52.190:45347: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:22.810655 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.805778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:40119)
0927 03:18:22.805815 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.805825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.805857 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.806001 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.806002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.806141 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.806195 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.806196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.807175 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.807176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.807718 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.807719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.808260 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.808261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.808324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.808820 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.808824 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.810610 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:18:22.810815 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.805817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:40119 (local address 127.17.52.190:45347)
0927 03:18:22.805857 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.805860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.805895 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:18:22.805919 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.805920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.805932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.805991 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.806275 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.807072 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.807802 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.808145 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.808235 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.810730 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:40119: BlockingRecv error: recv got EOF from 127.17.52.129:40119 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:22.815245 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.810915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:42069)
0927 03:18:22.811037 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.811048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.811096 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.811245 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.811246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.811353 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.811378 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.811379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.812255 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.812256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.812781 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.812783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.813216 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.813216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.813251 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.813643 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.813647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.815200 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260927 03:18:22.815356 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.811013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:42069 (local address 127.17.52.190:45347)
0927 03:18:22.811053 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.811056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.811083 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:18:22.811146 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.811148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.811161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.811229 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.811443 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.812186 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.812848 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.813146 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.813220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.815297 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:42069: BlockingRecv error: recv got EOF from 127.17.52.129:42069 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:18:22.815382 26497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:18:22.884639 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:22.884850 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:22.884887 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:22.886023 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:22.886085 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:22.886130 26501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:22.887585 26501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.130.keytab
--principal=oryx/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:36719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=40063
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:22.887828 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:22.888063 26501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:22.892760 26507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:22.892802 26508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.892946 26501 server_base.cc:1034] running on GCE node
W20260927 03:18:22.892971 26510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:22.893185 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:22.893435 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:22.894577 26501 hybrid_clock.cc:648] HybridClock initialized: now 1790479102894567 us; error 26 us; skew 500 ppm
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479102, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:22.897007 26501 init.cc:377] Logged in from keytab as oryx/127.17.52.130@KRBTEST.COM (short username oryx)
I20260927 03:18:22.897424 26501 webserver.cc:460] Webserver started at http://127.17.52.130:40063/ using document root <none> and password file <none>
I20260927 03:18:22.897657 26501 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:22.897734 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:22.899065 26501 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:22.899608 26517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.899792 26501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.899874 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "8102e87d5ce449e98de6b7e00ddcb8e9"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "1f7cbed605cb29fd36b97cc54c1c984e"
server_key_iv: "36e0ea6d2e0ef8779c61cd9887d31bc2"
server_key_version: "encryptionkey@0"
I20260927 03:18:22.900192 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:22.912933 26501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:22.915091 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:22.915248 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:22.915465 26501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:22.915760 26501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:22.915809 26501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.915850 26501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:22.915879 26501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:22.921998 26501 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:36719
I20260927 03:18:22.922035 26630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:36719 every 8 connection(s)
I20260927 03:18:22.922372 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:22.925820 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26501
I20260927 03:18:22.926206 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:36159
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:45347
--webserver_port=39035
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:18:22.929934 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.923048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:33967 (local address 127.17.52.190:45347)
0927 03:18:22.923138 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.923141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.923321 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:18:22.923774 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.923775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.923789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.923837 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.924772 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.925259 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.926275 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.926584 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.926661 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.929886 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:33967: BlockingRecv error: recv got EOF from 127.17.52.130:33967 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:18:22.930017 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.923031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:33967)
0927 03:18:22.923181 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.923213 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.923687 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.923895 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.923898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.924438 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.924680 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.924685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.925330 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.925331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.926194 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.926196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.926664 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.926665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.926767 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.927312 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.927320 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.929788 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
W20260927 03:18:22.930387 26631 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:45347 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.52.190:45347: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:22.934799 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.930609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:34631)
0927 03:18:22.930655 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.930685 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.930727 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.930862 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.930863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.930993 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.931033 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.931035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.931638 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.931639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.932144 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.932145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.932507 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.932508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.932593 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.933010 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.933015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.934700 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:18:22.934823 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.930671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:34631 (local address 127.17.52.190:45347)
0927 03:18:22.930719 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.930721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.930730 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:22.930803 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.930804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.930813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.930862 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.931105 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.931567 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.932235 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.932429 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.932506 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.934782 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:34631: BlockingRecv error: recv got EOF from 127.17.52.130:34631 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:22 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:22.939347 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:22.935107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:52771)
0927 03:18:22.935148 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:22.935157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:22.935194 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:22.935304 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:22.935305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:22.935450 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:22.935487 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.935489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.936048 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.936048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.936978 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:22.936980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.937279 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.937280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:22.937333 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.937668 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:22.937673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:22.939267 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:18:22.939394 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:22.935163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:52771 (local address 127.17.52.190:45347)
0927 03:18:22.935196 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:22.935197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:22.935206 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:22.935233 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:22.935233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:22.935240 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:22.935296 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:22.935555 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.935979 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.937050 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:22.937216 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:22.937266 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:22.939333 (+ 2067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:52771: BlockingRecv error: recv got EOF from 127.17.52.130:52771 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:18:22.939457 26631 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:18:23.009588 26634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:23.009831 26634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:23.009871 26634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:23.011036 26634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:23.011098 26634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:23.011144 26634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:23.012609 26634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/krb5kdc/oryx_127.17.52.131.keytab
--principal=oryx/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:36159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=39035
--tserver_master_addrs=127.17.52.190:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:23.012866 26634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:23.013090 26634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:23.017789 26640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:23.017756 26639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:23.017778 26642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:23.017812 26634 server_base.cc:1034] running on GCE node
I20260927 03:18:23.018208 26634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:23.018399 26634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:23.019564 26634 hybrid_clock.cc:648] HybridClock initialized: now 1790479103019552 us; error 27 us; skew 500 ppm
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479103, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:23.022066 26634 init.cc:377] Logged in from keytab as oryx/127.17.52.131@KRBTEST.COM (short username oryx)
I20260927 03:18:23.022467 26634 webserver.cc:460] Webserver started at http://127.17.52.131:39035/ using document root <none> and password file <none>
I20260927 03:18:23.022679 26634 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:23.022754 26634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:23.024027 26634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:23.025161 26649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:23.025324 26634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:23.025415 26634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "ac68285d8f3f454c9e1b12e844bd60cd"
format_stamp: "Formatted at 2026-09-27 03:18:22 on dist-test-slave-7j43"
server_key: "69e74b7a28a3cc55fa9203e2652ae30e"
server_key_iv: "9527a2df362890ed343f34b29279574b"
server_key_version: "encryptionkey@0"
I20260927 03:18:23.025732 26634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:23.033397 26634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:23.035444 26634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:23.035543 26634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:23.035737 26634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:23.036015 26634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:23.036047 26634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:23.036084 26634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:23.036110 26634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:23.041752 26634 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:36159
I20260927 03:18:23.041796 26762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:36159 every 8 connection(s)
I20260927 03:18:23.042128 26634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:18:23.049011 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:23.042849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:60381 (local address 127.17.52.190:45347)
0927 03:18:23.042955 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.042957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.043131 (+ 174us) server_negotiation.cc:407] Connection header received
0927 03:18:23.043569 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.043570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.043580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.043630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.044472 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.045030 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.045945 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.046142 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.046198 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.048967 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:60381: BlockingRecv error: recv got EOF from 127.17.52.131:60381 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 03:18:23.049031 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.042796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:60381)
0927 03:18:23.043020 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:23.043051 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:23.043482 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:23.043686 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:23.043689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:23.044228 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:23.044398 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.044403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.045107 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.045108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.045873 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.045875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.046209 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.046210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.046302 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.046709 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:23.046717 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:23.048900 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260927 03:18:23.049252 26763 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:45347 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.52.190:45347: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
I20260927 03:18:23.051681 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26634
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:23.053602 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.049472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:58513)
0927 03:18:23.049532 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:23.049543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:23.049587 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:23.049809 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:23.049810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:23.049915 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:23.049955 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.049956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.050505 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.050505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.050987 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.050989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.051263 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.051263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.051310 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.051688 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:23.051692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:23.053562 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 03:18:23.053701 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:23.049676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:58513 (local address 127.17.52.190:45347)
0927 03:18:23.049711 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.049713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.049719 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:18:23.049735 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.049736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.049743 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.049788 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.050012 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.050448 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.051046 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.051210 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.051255 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.053650 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:58513: BlockingRecv error: recv got EOF from 127.17.52.131:58513 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:23.057709 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.053819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:56297)
0927 03:18:23.053852 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:23.053861 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:23.053895 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:23.053998 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:23.053999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:23.054133 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:23.054158 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.054159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.054706 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.054707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.055167 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.055168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.055600 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.055601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.055638 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.055975 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:23.055983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:23.057629 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:18:23.057737 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:23.053875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:56297 (local address 127.17.52.190:45347)
0927 03:18:23.053910 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.053912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.053919 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:23.053940 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.053941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.053949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.053995 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.054221 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.054635 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.055245 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.055523 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.055584 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.057684 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:56297: BlockingRecv error: recv got EOF from 127.17.52.131:56297 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:18:23.057883 26763 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:18:23.059043 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.052043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.17.52.190:45347)
0927 03:18:23.052144 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.052146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.052270 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:18:23.052335 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.052337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.052348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.052416 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.053126 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.053823 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.054691 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.054896 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.054973 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.056146 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:23.056150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:23.056150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:23.056161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:23.057977 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:23.058284 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:23.058287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:23.058287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:23.058374 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:23.058528 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:23.058528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:23.058529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:23.058720 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:23.058825 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:23.058910 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:18:23.058986 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":68,"threads_started":1}
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:23.820012 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.815823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:51163)
0927 03:18:23.815949 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:23.815959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:23.816006 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:23.816132 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:23.816132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:23.816292 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:23.816339 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.816341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.816970 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.816971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.817532 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.817534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.817841 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.817841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.817878 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.818339 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:23.818344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:23.819946 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
W20260927 03:18:23.820108 26769 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:23.815867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:51163 (local address 127.17.52.190:45347)
0927 03:18:23.816009 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.816011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.816022 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:23.816047 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.816048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.816057 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.816120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.816403 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.816914 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.817600 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.817785 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.817832 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.820035 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:51163: BlockingRecv error: recv got EOF from 127.17.52.129:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:23 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:23.944326 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:23.939852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:41099)
0927 03:18:23.940083 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:23.940093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:23.940147 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:23.940258 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:23.940259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:23.940416 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:23.940490 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.940492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.941321 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.941322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.941857 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:23.941859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.942156 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.942156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:23.942194 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.942613 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:23.942618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:23.944251 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":175,"threads_started":1}
W20260927 03:18:23.944424 26769 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:23.939928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41099 (local address 127.17.52.190:45347)
0927 03:18:23.939992 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:23.939994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:23.940146 (+ 152us) server_negotiation.cc:407] Connection header received
0927 03:18:23.940193 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:23.940193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:23.940204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:23.940254 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:23.940585 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.941247 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.941920 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:23.942100 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:23.942148 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:23.944335 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:41099: BlockingRecv error: recv got EOF from 127.17.52.130:41099 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:24.062736 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:24.058271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:36859)
0927 03:18:24.058387 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:24.058397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:24.058480 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:24.058679 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:24.058680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:24.058890 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:24.058941 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.058943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.059559 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.059560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.060098 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.060099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.060395 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.060396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.060435 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.060896 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:24.060902 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:24.062627 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260927 03:18:24.062781 26769 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:24.058321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:36859 (local address 127.17.52.190:45347)
0927 03:18:24.058372 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:24.058374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:24.058487 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:18:24.058563 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:24.058564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:24.058574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:24.058653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:24.059004 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.059492 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.060161 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.060340 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.060387 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.062698 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:36859: BlockingRecv error: recv got EOF from 127.17.52.131:36859 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:24.824882 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:24.820445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:45279)
0927 03:18:24.820655 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:24.820665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:24.820726 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:24.820861 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:24.820861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:24.821012 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:24.821059 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.821061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.821784 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.821785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.822330 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.822332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.822641 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.822642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.822682 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.823083 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:24.823089 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:24.824826 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
W20260927 03:18:24.825050 26773 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:24.820486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:45279 (local address 127.17.52.190:45347)
0927 03:18:24.820645 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:24.820647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:24.820720 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:18:24.820776 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:24.820777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:24.820789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:24.820847 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:24.821112 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.821640 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.822405 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.822587 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.822631 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.824971 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:45279: BlockingRecv error: recv got EOF from 127.17.52.129:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:24 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:24.949419 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:24.944815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:60227)
0927 03:18:24.944956 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:24.944972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:24.945028 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:24.945154 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:24.945155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:24.945327 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:24.945373 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.945375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.945983 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.945984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.946510 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:24.946511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.946945 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.946945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:24.946987 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.947384 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:24.947389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:24.949326 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260927 03:18:24.949455 26773 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:24.944888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:60227 (local address 127.17.52.190:45347)
0927 03:18:24.944952 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:24.944954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:24.945024 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:18:24.945073 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:24.945073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:24.945083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:24.945138 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:24.945433 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.945916 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.946604 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:24.946884 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:24.946941 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:24.949397 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:60227: BlockingRecv error: recv got EOF from 127.17.52.130:60227 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:25.067720 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:25.063211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:44275)
0927 03:18:25.063387 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:25.063399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:25.063450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:25.063580 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:25.063581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:25.063744 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:25.063800 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.063801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.064454 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.064455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.064998 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.064999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.065304 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.065305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.065343 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.065777 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:25.065781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:25.067646 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
W20260927 03:18:25.067813 26773 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:25.063346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:44275 (local address 127.17.52.190:45347)
0927 03:18:25.063431 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:25.063433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:25.063444 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:25.063495 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:25.063496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:25.063508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:25.063563 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:25.063874 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.064390 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.065061 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.065240 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.065305 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.067745 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:44275: BlockingRecv error: recv got EOF from 127.17.52.131:44275 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:25.832829 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:25.825252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:47975)
0927 03:18:25.825406 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:25.825416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:25.825463 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:25.825586 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:25.825586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:25.825739 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:25.825790 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.825791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.826423 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.826423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.826931 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.826933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.827224 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.827224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.827261 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.830799 (+ 3538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:25.830803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:25.832764 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260927 03:18:25.833014 26777 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:25.825321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47975 (local address 127.17.52.190:45347)
0927 03:18:25.825440 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:25.825442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:25.825452 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:25.825508 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:25.825509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:25.825519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:25.825573 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:25.825852 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.826360 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.826989 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.827170 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.827218 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.832914 (+ 5696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:47975: BlockingRecv error: recv got EOF from 127.17.52.129:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:25 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:25.954476 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:25.949864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:49969)
0927 03:18:25.950025 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:25.950035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:25.950107 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:25.950256 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:25.950257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:25.950440 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:25.950485 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.950487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.951148 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.951149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.951680 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:25.951681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.952044 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.952045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:25.952104 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.952523 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:25.952527 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:25.954382 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
W20260927 03:18:25.954525 26777 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:25.949921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:49969 (local address 127.17.52.190:45347)
0927 03:18:25.949985 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:25.949987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:25.950095 (+ 108us) server_negotiation.cc:407] Connection header received
0927 03:18:25.950164 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:25.950165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:25.950180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:25.950249 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:25.950561 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.951066 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.951762 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:25.951964 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:25.952042 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:25.954461 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:49969: BlockingRecv error: recv got EOF from 127.17.52.130:49969 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:26.072556 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:26.068140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:54579)
0927 03:18:26.068277 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:26.068287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:26.068341 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:26.068468 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:26.068469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:26.068666 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:26.068713 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.068715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.069376 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.069377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.069913 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.069914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.070233 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.070234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.070279 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.070736 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:26.070741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:26.072500 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
W20260927 03:18:26.072700 26777 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:26.068206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:54579 (local address 127.17.52.190:45347)
0927 03:18:26.068258 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:26.068260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:26.068341 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:18:26.068398 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:26.068399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:26.068410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:26.068451 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:26.068798 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.069307 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.069988 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.070173 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.070229 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.072634 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:54579: BlockingRecv error: recv got EOF from 127.17.52.131:54579 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:26.837536 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:26.833235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:43443)
0927 03:18:26.833363 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:26.833379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:26.833428 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:26.833556 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:26.833556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:26.833710 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:26.833758 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.833759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.834383 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.834384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.834892 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.834893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.835193 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.835194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.835231 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.835656 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:26.835660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:26.837443 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260927 03:18:26.837577 26781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:26.833275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:43443 (local address 127.17.52.190:45347)
0927 03:18:26.833400 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:26.833402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:26.833413 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:26.833481 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:26.833482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:26.833492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:26.833541 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:26.833815 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.834322 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.834954 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.835137 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.835185 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.837508 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:43443: BlockingRecv error: recv got EOF from 127.17.52.129:43443 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:26 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:26.959247 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:26.955068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:37167)
0927 03:18:26.955201 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:26.955211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:26.955289 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:26.955419 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:26.955420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:26.955572 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:26.955619 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.955620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.956227 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.956228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.956768 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:26.956770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.957060 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.957061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:26.957098 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.957529 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:26.957533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:26.959192 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260927 03:18:26.959358 26781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:26.955120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:37167 (local address 127.17.52.190:45347)
0927 03:18:26.955184 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:26.955186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:26.955266 (+ 80us) server_negotiation.cc:407] Connection header received
0927 03:18:26.955342 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:26.955343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:26.955353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:26.955403 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:26.955673 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.956168 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.956827 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:26.957004 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:26.957054 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:26.959286 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:37167: BlockingRecv error: recv got EOF from 127.17.52.130:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:27.077399 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:27.073031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:54091)
0927 03:18:27.073145 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:27.073157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:27.073219 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:27.073347 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:27.073348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:27.073504 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:27.073551 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.073552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.074166 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.074167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.074674 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.074675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.074981 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.074982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.075024 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.075456 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:27.075461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:27.077287 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
W20260927 03:18:27.077533 26781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:27.073117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:54091 (local address 127.17.52.190:45347)
0927 03:18:27.073164 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:27.073166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:27.073205 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:18:27.073278 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:27.073279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:27.073291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:27.073332 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:27.073612 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.074105 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.074739 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.074918 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.074968 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.077404 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:54091: BlockingRecv error: recv got EOF from 127.17.52.131:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:27.842399 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:27.838019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:33527)
0927 03:18:27.838165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:27.838178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:27.838269 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:27.838435 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:27.838436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:27.838632 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:27.838680 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.838682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.839323 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.839323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.839841 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.839842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.840145 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.840146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.840184 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.840683 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:27.840688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:27.842341 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
W20260927 03:18:27.842511 26785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:27.838089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:33527 (local address 127.17.52.190:45347)
0927 03:18:27.838185 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:27.838187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:27.838264 (+ 77us) server_negotiation.cc:407] Connection header received
0927 03:18:27.838336 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:27.838337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:27.838349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:27.838416 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:27.838747 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.839253 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.839906 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.840088 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.840135 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.842430 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:33527: BlockingRecv error: recv got EOF from 127.17.52.129:33527 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:27 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:27.964428 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:27.959734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:54371)
0927 03:18:27.959899 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:27.959912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:27.959976 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:27.960105 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:27.960106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:27.960296 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:27.960341 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.960343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.961103 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.961105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.961693 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:27.961695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.962071 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.962072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:27.962130 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.962606 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:27.962611 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:27.964334 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
W20260927 03:18:27.964478 26785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:27.959863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:54371 (local address 127.17.52.190:45347)
0927 03:18:27.959942 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:27.959944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:27.959956 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:18:27.960036 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:27.960037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:27.960049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:27.960102 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:27.960424 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.961015 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.961781 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:27.961988 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:27.962070 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:27.964413 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:54371: BlockingRecv error: recv got EOF from 127.17.52.130:54371 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:28.082335 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:28.077808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:42359)
0927 03:18:28.077959 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:28.077968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:28.078046 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:28.078202 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:28.078203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:28.078380 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:28.078427 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.078429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.079088 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.079090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.079619 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.079620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.080000 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.080001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.080062 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.080481 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:28.080485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:28.082277 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260927 03:18:28.082446 26785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:28.077857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:42359 (local address 127.17.52.190:45347)
0927 03:18:28.077919 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:28.077922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:28.078027 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:18:28.078113 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:28.078114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:28.078129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:28.078196 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:28.078501 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.079009 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.079713 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.079909 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.080001 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.082378 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:42359: BlockingRecv error: recv got EOF from 127.17.52.131:42359 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:28.847066 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:28.842883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:47023)
0927 03:18:28.843015 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:28.843026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:28.843080 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:28.843209 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:28.843209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:28.843356 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:28.843401 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.843402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.844020 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.844021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.844521 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.844522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.844871 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.844872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.844913 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.845313 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:28.845318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:28.847010 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260927 03:18:28.847285 26789 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:28.842972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47023 (local address 127.17.52.190:45347)
0927 03:18:28.843098 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:28.843099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:28.843108 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:28.843128 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:28.843128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:28.843139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:28.843195 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:28.843464 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.843962 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.844620 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.844814 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.844861 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.847185 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:47023: BlockingRecv error: recv got EOF from 127.17.52.129:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:28 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:28.969606 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:28.964912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:41541)
0927 03:18:28.965092 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:28.965105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:28.965178 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:28.965347 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:28.965348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:28.965568 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:28.965624 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.965626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.966423 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.966424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.966965 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:28.966966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.967297 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.967298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:28.967336 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.967752 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:28.967756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:28.969511 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
W20260927 03:18:28.969651 26789 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:28.965024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41541 (local address 127.17.52.190:45347)
0927 03:18:28.965096 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:28.965099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:28.965159 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:18:28.965235 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:28.965236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:28.965248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:28.965324 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:28.965701 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.966356 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.967033 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:28.967219 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:28.967283 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:28.969588 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:41541: BlockingRecv error: recv got EOF from 127.17.52.130:41541 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:29.087447 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:29.082871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:56561)
0927 03:18:29.082997 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:29.083015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:29.083099 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:29.083274 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:29.083274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:29.083486 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:29.083547 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.083549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.084309 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.084310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.084875 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.084876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.085248 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.085249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.085313 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.085717 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:29.085721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:29.087390 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260927 03:18:29.087580 26789 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:29.082919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:56561 (local address 127.17.52.190:45347)
0927 03:18:29.082985 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:29.082987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:29.083077 (+ 90us) server_negotiation.cc:407] Connection header received
0927 03:18:29.083179 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:29.083180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:29.083195 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:29.083251 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:29.083627 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.084217 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.084959 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.085161 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.085247 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.087494 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:56561: BlockingRecv error: recv got EOF from 127.17.52.131:56561 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:29.851991 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:29.847469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:59911)
0927 03:18:29.847633 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:29.847644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:29.847693 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:29.847899 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:29.847900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:29.848053 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:29.848102 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.848103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.848780 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.848781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.849365 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.849366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.849686 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.849687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.849727 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.850211 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:29.850221 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:29.851932 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260927 03:18:29.852105 26793 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:29.847558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59911 (local address 127.17.52.190:45347)
0927 03:18:29.847672 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:29.847674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:29.847683 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:18:29.847799 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:29.847799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:29.847812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:29.847880 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:29.848163 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.848682 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.849432 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.849625 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.849676 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.852024 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:59911: BlockingRecv error: recv got EOF from 127.17.52.129:59911 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:29 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:29.974570 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:29.970094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:43949)
0927 03:18:29.970242 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:29.970253 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:29.970337 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:29.970451 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:29.970451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:29.970611 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:29.970660 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.970662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.971288 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.971289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.971807 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:29.971809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.972121 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.972122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:29.972159 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.972625 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:29.972630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:29.974482 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
W20260927 03:18:29.974664 26793 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:29.970155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:43949 (local address 127.17.52.190:45347)
0927 03:18:29.970251 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:29.970252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:29.970322 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:18:29.970385 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:29.970386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:29.970396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:29.970440 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:29.970718 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.971207 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.971879 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:29.972060 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:29.972111 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:29.974599 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:43949: BlockingRecv error: recv got EOF from 127.17.52.130:43949 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:30.092283 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:30.087903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:42347)
0927 03:18:30.088059 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:30.088069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:30.088123 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:30.088257 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:30.088258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:30.088426 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:30.088473 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.088475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.089112 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.089113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.089653 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.089654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.089985 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.089987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.090043 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.090472 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:30.090477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:30.092212 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
W20260927 03:18:30.092348 26793 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:30.087969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:42347 (local address 127.17.52.190:45347)
0927 03:18:30.088040 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:30.088042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:30.088124 (+ 82us) server_negotiation.cc:407] Connection header received
0927 03:18:30.088181 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:30.088181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:30.088193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:30.088235 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:30.088534 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.089048 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.089721 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.089894 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.089944 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.092303 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:42347: BlockingRecv error: recv got EOF from 127.17.52.131:42347 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:30.860795 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:30.852491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:59051)
0927 03:18:30.852798 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:30.852808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:30.852866 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:30.852996 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:30.852997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:30.853154 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:30.853200 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.853202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.853837 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.853837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.854353 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.854355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.854688 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.854688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.854727 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.858666 (+ 3939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:30.858672 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:30.860681 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
W20260927 03:18:30.860881 26797 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:30.852639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59051 (local address 127.17.52.190:45347)
0927 03:18:30.852769 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:30.852771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:30.852869 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:18:30.852909 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:30.852910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:30.852920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:30.852978 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:30.853262 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.853771 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.854422 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.854625 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.854679 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.860807 (+ 6128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:59051: BlockingRecv error: recv got EOF from 127.17.52.129:59051 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:30 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:30.980047 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:30.975040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:41375)
0927 03:18:30.975186 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:30.975200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:30.975279 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:30.975415 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:30.975416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:30.975582 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:30.975632 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.975634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.976491 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.976492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.977129 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:30.977132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.977518 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.977519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:30.977565 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.978056 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:30.978061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:30.979932 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
W20260927 03:18:30.980108 26797 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:30.975134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41375 (local address 127.17.52.190:45347)
0927 03:18:30.975198 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:30.975200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:30.975270 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:18:30.975338 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:30.975338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:30.975348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:30.975396 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:30.975722 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.976395 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.977212 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:30.977409 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:30.977456 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:30.980019 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:41375: BlockingRecv error: recv got EOF from 127.17.52.130:41375 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:31.097276 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:31.092856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:56921)
0927 03:18:31.093016 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:31.093026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:31.093088 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:31.093221 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:31.093222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:31.093384 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:31.093431 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.093432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.094061 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.094061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.094578 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.094579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.094919 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.094919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.094958 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.095370 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:31.095374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:31.097201 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
W20260927 03:18:31.097424 26797 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:31.092912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:56921 (local address 127.17.52.190:45347)
0927 03:18:31.092968 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:31.092971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:31.093094 (+ 123us) server_negotiation.cc:407] Connection header received
0927 03:18:31.093142 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:31.093143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:31.093158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:31.093204 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:31.093494 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.093994 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.094660 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.094851 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.094909 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.097331 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:56921: BlockingRecv error: recv got EOF from 127.17.52.131:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:31.865994 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:31.861373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:60563)
0927 03:18:31.861525 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:31.861535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:31.861598 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:31.861732 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:31.861733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:31.861899 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:31.861946 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.861948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.862616 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.862617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.863128 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.863130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.863464 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.863465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.863503 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.864051 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:31.864080 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:31.865929 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260927 03:18:31.866127 26801 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:31.861452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:60563 (local address 127.17.52.190:45347)
0927 03:18:31.861550 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:31.861552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:31.861589 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:18:31.861644 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:31.861645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:31.861655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:31.861713 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:31.862041 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.862543 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.863205 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.863400 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.863457 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.866017 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:60563: BlockingRecv error: recv got EOF from 127.17.52.129:60563 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:31 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:31.985785 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:31.980843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:34867)
0927 03:18:31.981007 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:31.981019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:31.981109 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:31.981283 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:31.981284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:31.981462 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:31.981508 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.981510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.982252 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.982253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.982862 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:31.982864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.983291 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.983292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:31.983351 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.983751 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:31.983755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:31.985717 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
W20260927 03:18:31.985886 26801 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:31.980899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:34867 (local address 127.17.52.190:45347)
0927 03:18:31.980970 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:31.980973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:31.981088 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:18:31.981192 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:31.981193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:31.981205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:31.981277 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:31.981588 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.982166 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.982941 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:31.983217 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:31.983294 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:31.985818 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:34867: BlockingRecv error: recv got EOF from 127.17.52.130:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:32.102759 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:32.097751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:53775)
0927 03:18:32.097901 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:32.097912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:32.097968 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:32.098096 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:32.098097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:32.098426 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:32.098475 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.098477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.099111 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.099111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.099666 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.099668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.100006 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.100007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.100059 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.100641 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:32.100646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:32.102701 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260927 03:18:32.102938 26801 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:32.097880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:53775 (local address 127.17.52.190:45347)
0927 03:18:32.097945 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:32.097947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:32.097958 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:32.098025 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:32.098026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:32.098038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:32.098081 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:32.098537 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.099037 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.099736 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.099924 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.099974 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.102827 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:53775: BlockingRecv error: recv got EOF from 127.17.52.131:53775 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:32.871177 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:32.866444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:48681)
0927 03:18:32.866583 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:32.866597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:32.866648 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:32.866817 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:32.866818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:32.866975 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:32.867023 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.867025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.867687 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.867688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.868208 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.868209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.868536 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.868537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.868600 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.869214 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:32.869220 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:32.871114 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
W20260927 03:18:32.871326 26805 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:32.866528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:48681 (local address 127.17.52.190:45347)
0927 03:18:32.866685 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:32.866687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:32.866695 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:32.866717 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:32.866717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:32.866730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:32.866801 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:32.867092 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.867585 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.868281 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.868472 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.868524 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.871227 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:48681: BlockingRecv error: recv got EOF from 127.17.52.129:48681 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:32 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:32.994716 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:32.986360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:33287)
0927 03:18:32.986516 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:32.986525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:32.986582 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:32.986713 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:32.986713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:32.986881 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:32.986928 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.986930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.987549 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.987550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.988072 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:32.988073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.988400 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.988401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:32.988459 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.992847 (+ 4388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:32.992854 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:32.994610 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260927 03:18:32.994774 26805 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:32.986436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:33287 (local address 127.17.52.190:45347)
0927 03:18:32.986526 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:32.986528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:32.986588 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:18:32.986629 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:32.986630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:32.986641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:32.986698 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:32.986993 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.987484 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.988143 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:32.988328 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:32.988378 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:32.994698 (+ 6320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:33287: BlockingRecv error: recv got EOF from 127.17.52.130:33287 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:33.107916 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:33.103285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:49995)
0927 03:18:33.103443 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:33.103453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:33.103499 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:33.103627 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:33.103627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:33.103781 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:33.103826 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.103827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.104473 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.104474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.105045 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.105047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.105377 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.105378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.105416 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:33.105882 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:33.105886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:33.107825 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
W20260927 03:18:33.107946 26805 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:33.103386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:49995 (local address 127.17.52.190:45347)
0927 03:18:33.103484 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:33.103486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:33.103496 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:33.103542 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:33.103543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:33.103554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:33.103611 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:33.103905 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.104407 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.105118 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.105313 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.105371 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:33.107889 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:49995: BlockingRecv error: recv got EOF from 127.17.52.131:49995 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:33.876365 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:33.871661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:58367)
0927 03:18:33.871838 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:33.871851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:33.871923 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:33.872067 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:33.872068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:33.872243 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:33.872305 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.872307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.872974 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.872975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.873533 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.873535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.873865 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.873866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.873907 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:33.874351 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:33.874355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:33.876293 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
W20260927 03:18:33.876504 26809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:33.871792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:58367 (local address 127.17.52.190:45347)
0927 03:18:33.871931 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:33.871933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:33.871960 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:18:33.871981 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:33.871982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:33.871991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:33.872048 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:33.872372 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.872905 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.873606 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.873801 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.873852 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:33.876391 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:58367: BlockingRecv error: recv got EOF from 127.17.52.129:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:33 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:34.000538 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:33.995205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:33943)
0927 03:18:33.995398 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:33.995414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:33.995511 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:33.995729 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:33.995730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:33.995924 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:33.995970 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.995972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.996740 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.996742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.997568 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:33.997571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.997968 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.997969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:33.998026 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:33.998493 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:33.998497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:34.000420 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
W20260927 03:18:34.000627 26809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:33.995330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:33943 (local address 127.17.52.190:45347)
0927 03:18:33.995427 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:33.995429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:33.995500 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:18:33.995623 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:33.995624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:33.995636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:33.995718 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:33.996070 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.996643 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.997658 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:33.997861 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:33.997970 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:34.000513 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:33943: BlockingRecv error: recv got EOF from 127.17.52.130:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 03:18:34 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:34 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:34.113044 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:34.108377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:43609)
0927 03:18:34.108544 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:34.108556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:34.108647 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:34.108799 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:34.108800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:34.108978 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:34.109021 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:34.109023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.109697 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.109698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:34.110236 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:34.110237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.110610 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.110611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:34.110666 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:34.111152 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:34.111164 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:34.112940 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
W20260927 03:18:34.113077 26809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:34.108456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:43609 (local address 127.17.52.190:45347)
0927 03:18:34.108535 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:34.108538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:34.108641 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:18:34.108711 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:34.108712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:34.108725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:34.108776 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:34.109098 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.109614 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.110328 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.110525 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.110608 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:34.113014 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:43609: BlockingRecv error: recv got EOF from 127.17.52.131:43609 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 03:18:34 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:34 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:34.881700 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:34.876766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:43793)
0927 03:18:34.876987 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:34.876999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:34.877068 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:34.877249 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:34.877250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:34.877429 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:34.877476 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:34.877478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.878129 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.878130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:34.878657 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:34.878659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.879050 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.879051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:34.879099 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:34.879713 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:34.879718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:34.881605 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
W20260927 03:18:34.881752 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:34.876855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:43793 (local address 127.17.52.190:45347)
0927 03:18:34.876961 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:34.876965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:34.877066 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:18:34.877129 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:34.877130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:34.877140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:34.877188 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:34.877543 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.878045 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.878727 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:34.878927 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:34.878981 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:34.881680 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:43793: BlockingRecv error: recv got EOF from 127.17.52.129:43793 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:35.006446 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:35.001035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:38209)
0927 03:18:35.001199 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:35.001212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:35.001287 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:35.001486 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:35.001487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:35.001660 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:35.001707 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.001709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.002535 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.002536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.003333 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.003335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.003709 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.003710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.003768 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.004199 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:35.004203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:35.006343 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
W20260927 03:18:35.006485 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:35.001141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:38209 (local address 127.17.52.190:45347)
0927 03:18:35.001211 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:35.001213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:35.001282 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:18:35.001344 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:35.001345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:35.001357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:35.001405 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:35.001787 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.002470 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.003411 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.003629 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.003700 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.006423 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:38209: BlockingRecv error: recv got EOF from 127.17.52.130:38209 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:35.117756 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:35.113470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:40485)
0927 03:18:35.113603 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:35.113636 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:35.113690 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:35.113817 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:35.113818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:35.113977 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:35.114022 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.114024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.114645 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.114646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.115172 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.115173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.115467 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.115467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.115506 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.115942 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:35.115947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:35.117698 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
W20260927 03:18:35.117888 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:35.113543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:40485 (local address 127.17.52.190:45347)
0927 03:18:35.113605 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:35.113607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:35.113688 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:18:35.113732 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:35.113732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:35.113743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:35.113797 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:35.114077 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.114558 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.115235 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.115407 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.115455 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.117803 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:40485: BlockingRecv error: recv got EOF from 127.17.52.131:40485 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:35 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:35.890486 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:35.882142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:44811)
0927 03:18:35.882259 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:35.882280 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:35.882356 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:35.882485 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:35.882486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:35.882649 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:35.882694 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.882696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.883439 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.883439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.884009 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:35.884011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.884335 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.884335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:35.884376 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.888450 (+ 4074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:35.888456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:35.890423 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
W20260927 03:18:35.890641 26817 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:35.882207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:44811 (local address 127.17.52.190:45347)
0927 03:18:35.882382 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:35.882384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:35.882392 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:35.882414 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:35.882415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:35.882425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:35.882482 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:35.882779 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.883325 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.884077 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:35.884266 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:35.884323 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:35.890543 (+ 6220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:44811: BlockingRecv error: recv got EOF from 127.17.52.129:44811 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:36.011622 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:36.006938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:42607)
0927 03:18:36.007068 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:36.007099 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:36.007175 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:36.007322 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:36.007323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:36.007508 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:36.007552 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.007554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.008242 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.008244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.008857 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.008859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.009248 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.009249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.009310 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.009777 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:36.009781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:36.011512 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260927 03:18:36.011700 26817 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:36.007068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:42607 (local address 127.17.52.190:45347)
0927 03:18:36.007139 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:36.007141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:36.007157 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:18:36.007233 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:36.007234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:36.007249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:36.007317 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:36.007666 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.008149 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.008940 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.009137 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.009229 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.011602 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:42607: BlockingRecv error: recv got EOF from 127.17.52.130:42607 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:36.122514 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:36.118205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:45241)
0927 03:18:36.118369 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:36.118379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:36.118432 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:36.118538 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:36.118539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:36.118695 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:36.118741 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.118742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.119352 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.119353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.119875 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.119877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.120182 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.120183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.120221 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.120743 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:36.120748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:36.122457 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
W20260927 03:18:36.122650 26817 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:36.118261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:45241 (local address 127.17.52.190:45347)
0927 03:18:36.118341 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:36.118342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:36.118432 (+ 90us) server_negotiation.cc:407] Connection header received
0927 03:18:36.118472 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:36.118473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:36.118484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:36.118527 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:36.118800 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.119287 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.119942 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.120120 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.120172 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.122568 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:45241: BlockingRecv error: recv got EOF from 127.17.52.131:45241 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:36 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:36.895574 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:36.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:51089)
0927 03:18:36.891077 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:36.891089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:36.891146 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:36.891309 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:36.891310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:36.891476 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:36.891531 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.891532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.892174 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.892175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.892776 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:36.892778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.893138 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.893139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:36.893238 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.893702 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:36.893707 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:36.895504 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260927 03:18:36.895682 26821 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:36.891020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:51089 (local address 127.17.52.190:45347)
0927 03:18:36.891148 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:36.891151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:36.891198 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:18:36.891224 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:36.891225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:36.891234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:36.891289 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:36.891597 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.892104 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.892848 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:36.893041 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:36.893097 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:36.895603 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:51089: BlockingRecv error: recv got EOF from 127.17.52.129:51089 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:37.016371 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:37.012095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:39809)
0927 03:18:37.012225 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:37.012237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:37.012303 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:37.012428 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:37.012429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:37.012617 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:37.012665 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.012666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.013299 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.013300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.013810 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.013811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.014129 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.014129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.014166 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.014607 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:37.014611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:37.016310 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260927 03:18:37.016515 26821 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:37.012196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:39809 (local address 127.17.52.190:45347)
0927 03:18:37.012247 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:37.012248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:37.012290 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:18:37.012359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:37.012359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:37.012369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:37.012410 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:37.012733 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.013224 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.013885 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.014067 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.014128 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.016428 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:39809: BlockingRecv error: recv got EOF from 127.17.52.130:39809 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.131@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:37.127297 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:37.122933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.131:33727)
0927 03:18:37.123057 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:37.123068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:37.123131 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:37.123238 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:37.123239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:37.123412 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:37.123456 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.123458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.124078 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.124078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.124641 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.124643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.124966 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.124967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.125007 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.125477 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:37.125482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:37.127218 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20260927 03:18:37.127410 26821 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:37.123007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:33727 (local address 127.17.52.190:45347)
0927 03:18:37.123058 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:37.123059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:37.123132 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:18:37.123173 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:37.123173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:37.123186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:37.123233 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:37.123514 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.124008 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.124708 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.124900 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.124953 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.127327 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.131:33727: BlockingRecv error: recv got EOF from 127.17.52.131:33727 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:37 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.129@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:37.900628 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:37.896129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.129:46361)
0927 03:18:37.896275 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:37.896287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:37.896354 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:37.896477 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:37.896478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:37.896653 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:37.896699 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.896701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.897351 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.897352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.897892 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:37.897893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.898225 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.898226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:37.898266 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.898679 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:37.898683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:37.900517 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
W20260927 03:18:37.900811 26825 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:37.896197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:46361 (local address 127.17.52.190:45347)
0927 03:18:37.896351 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:37.896353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:37.896361 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:18:37.896401 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:37.896402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:37.896412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:37.896459 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:37.896776 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.897279 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.897964 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:37.898159 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:37.898212 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:37.900705 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.129:46361: BlockingRecv error: recv got EOF from 127.17.52.129:46361 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
Sep 27 03:18:38 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:18:38 dist-test-slave-7j43 krb5kdc[25766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.52.130@KRBTEST.COM for oryx/127.17.52.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:18:38.021785 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.016868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45347 (local address 127.17.52.130:50335)
0927 03:18:38.017063 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:38.017081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:38.017165 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:38.017320 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:38.017321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:38.017494 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:38.017539 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.017541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.018215 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.018217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.018776 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.018778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.019166 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.019167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.019232 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.019682 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:38.019686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:38.021688 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.52.190:45347: Server oryx/127.17.52.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
W20260927 03:18:38.021835 26825 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:38.016933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:50335 (local address 127.17.52.190:45347)
0927 03:18:38.017026 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.017029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.017150 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:18:38.017252 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.017253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.017264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.017305 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.017628 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.018134 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.018867 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.019070 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.019167 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.021770 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.52.130:50335: BlockingRecv error: recv got EOF from 127.17.52.130:50335 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 03:18:38.053099 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26350
I20260927 03:18:38.057515 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26501
I20260927 03:18:38.061723 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26634
I20260927 03:18:38.065716 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26279
2026-09-27T03:18:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17398 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T03:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:38Z Disabled control of system clock
I20260927 03:18:38.092509 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43793
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:18:38.171801 26830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.172003 26830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.172022 26830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.173246 26830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:38.173293 26830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.173306 26830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.173319 26830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:38.173377 26830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:38.174674 26830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:43793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.174949 26830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.175271 26830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:38.180356 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.180403 26830 server_base.cc:1034] running on GCE node
W20260927 03:18:38.180507 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.180339 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.180807 26830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.181092 26830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.182235 26830 hybrid_clock.cc:648] HybridClock initialized: now 1790479118182224 us; error 31 us; skew 500 ppm
I20260927 03:18:38.183485 26830 webserver.cc:460] Webserver started at http://127.17.52.190:45717/ using document root <none> and password file <none>
I20260927 03:18:38.183669 26830 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.183710 26830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.183801 26830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.184857 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "84ad6b31e4bd4d40b080a7016b90b1f7"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "43b90df7476a349c13607e0f00c7c5cc"
server_key_iv: "349c6420621532a42bab9323ecf16aa8"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.185156 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "84ad6b31e4bd4d40b080a7016b90b1f7"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "43b90df7476a349c13607e0f00c7c5cc"
server_key_iv: "349c6420621532a42bab9323ecf16aa8"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.186383 26830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:38.187036 26844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.187208 26830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:38.187283 26830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "84ad6b31e4bd4d40b080a7016b90b1f7"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "43b90df7476a349c13607e0f00c7c5cc"
server_key_iv: "349c6420621532a42bab9323ecf16aa8"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.187330 26830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.209663 26830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.209924 26830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.210026 26830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.213948 26830 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:43793
I20260927 03:18:38.213979 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:43793 every 8 connection(s)
I20260927 03:18:38.214329 26830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:38.214970 26897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:38.216547 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26830
I20260927 03:18:38.216682 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:38.216876 17618 external_mini_cluster.cc:1428] Setting key 699327dd6d401eb6394a54252aedefe6
I20260927 03:18:38.217840 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7: Bootstrap starting.
I20260927 03:18:38.218540 26897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:38.218861 26897 log.cc:824] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:38.219583 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7: No bootstrap required, opened a new log
I20260927 03:18:38.221938 26897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } }
I20260927 03:18:38.222127 26897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:38.222170 26897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ad6b31e4bd4d40b080a7016b90b1f7, State: Initialized, Role: FOLLOWER
I20260927 03:18:38.222333 26897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } }
I20260927 03:18:38.222410 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.217408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.17.52.190:43793)
0927 03:18:38.217592 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.217599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.217623 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:18:38.218060 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.218067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.218130 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.218245 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.218475 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.219427 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.220488 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.220959 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.221048 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.221733 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.221737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.221737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.221901 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.221935 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.221947 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.221998 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.222110 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.222213 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260927 03:18:38.222429 26897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:38.222688 26897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:38.222731 26897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:38.223366 26897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } }
I20260927 03:18:38.223481 26897 leader_election.cc:302] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ad6b31e4bd4d40b080a7016b90b1f7; no voters:
I20260927 03:18:38.223651 26897 leader_election.cc:288] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:38.223726 26902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:38.223905 26902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [term 1 LEADER]: Becoming Leader. State: Replica: 84ad6b31e4bd4d40b080a7016b90b1f7, State: Running, Role: LEADER
I20260927 03:18:38.223919 26897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:38.224016 26902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } }
I20260927 03:18:38.224375 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } } }
I20260927 03:18:38.224381 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ad6b31e4bd4d40b080a7016b90b1f7. Latest consensus state: current_term: 1 leader_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ad6b31e4bd4d40b080a7016b90b1f7" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43793 } } }
I20260927 03:18:38.224557 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:38.224983 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:38.224996 26911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:38.225479 26911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:38.226997 26911 catalog_manager.cc:1347] Generated new cluster ID: dd43cbabe8a249f3a700ddd65658e8d4
I20260927 03:18:38.227041 26911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:38.244563 26911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:38.245115 26911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:38.249603 26911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ad6b31e4bd4d40b080a7016b90b1f7: Generated new TSK 0
I20260927 03:18:38.249778 26911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:38.256028 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--builtin_ntp_servers=127.17.52.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:18:38.354413 26921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.354616 26921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.354636 26921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.355803 26921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.355844 26921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.355870 26921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:38.357225 26921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.357482 26921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.357796 26921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:38.363554 26929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.363579 26927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.363574 26926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.363752 26921 server_base.cc:1034] running on GCE node
I20260927 03:18:38.364003 26921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.364260 26921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.365432 26921 hybrid_clock.cc:648] HybridClock initialized: now 1790479118365402 us; error 37 us; skew 500 ppm
I20260927 03:18:38.366935 26921 webserver.cc:460] Webserver started at http://127.17.52.129:34485/ using document root <none> and password file <none>
I20260927 03:18:38.367208 26921 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.367278 26921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.367405 26921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.368764 26921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "e6e3ed5941db452abd74e3b15543b240"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "c9c104559b9206f3ed0c375b8d9e9850"
server_key_iv: "5e2fbcdc75be0c3d1e125c6af51a6a2a"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.369134 26921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "e6e3ed5941db452abd74e3b15543b240"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "c9c104559b9206f3ed0c375b8d9e9850"
server_key_iv: "5e2fbcdc75be0c3d1e125c6af51a6a2a"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.370639 26921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:38.371467 26935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.371657 26921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:38.371742 26921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "e6e3ed5941db452abd74e3b15543b240"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "c9c104559b9206f3ed0c375b8d9e9850"
server_key_iv: "5e2fbcdc75be0c3d1e125c6af51a6a2a"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.371819 26921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.393847 26921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.394111 26921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.394212 26921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.394392 26921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:38.394729 26921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:38.394763 26921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.394829 26921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:38.394853 26921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.400715 26921 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:42757
I20260927 03:18:38.400755 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:42757 every 8 connection(s)
I20260927 03:18:38.401078 26921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:38.401528 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 26921
I20260927 03:18:38.401659 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:38.401877 17618 external_mini_cluster.cc:1428] Setting key e3eb2e7fb1b82cd9c7261d71a7b4b27a
I20260927 03:18:38.402860 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--builtin_ntp_servers=127.17.52.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:18:38.406996 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.401892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47991 (local address 127.17.52.190:43793)
0927 03:18:38.401947 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.401949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.402238 (+ 289us) server_negotiation.cc:407] Connection header received
0927 03:18:38.402859 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.402860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.402873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.402939 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.403720 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.404334 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.405555 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.405772 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.405856 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.406716 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.406718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.406718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.406723 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.406736 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.406740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.406786 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.406876 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.406919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:18:38.407226 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.401892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43793 (local address 127.17.52.129:47991)
0927 03:18:38.402120 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:38.402164 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:38.402753 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:38.403135 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:38.403139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:38.403149 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:38.403631 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.403635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.404424 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.404425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.405467 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.405469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.405865 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.405867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.405996 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.406551 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:38.406552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:38.406579 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:38.406601 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:38.406788 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:38.406793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:38.406796 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:38.406879 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:18:38.406991 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260927 03:18:38.407637 27049 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43793
I20260927 03:18:38.407722 27049 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:38.407913 27049 heartbeater.cc:511] Master 127.17.52.190:43793 requested a full tablet report, sending...
I20260927 03:18:38.408409 26861 ts_manager.cc:194] Registered new tserver with Master: e6e3ed5941db452abd74e3b15543b240 (127.17.52.129:42757)
I20260927 03:18:38.409046 26861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.129:47991
W20260927 03:18:38.486912 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.487231 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.487267 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.488613 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.488708 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.488752 27052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:38.490229 27052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.490495 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.490804 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:18:38.495786 27052 server_base.cc:1034] running on GCE node
W20260927 03:18:38.495807 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.495752 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.495750 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.496210 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.496477 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.497635 27052 hybrid_clock.cc:648] HybridClock initialized: now 1790479118497619 us; error 29 us; skew 500 ppm
I20260927 03:18:38.498947 27052 webserver.cc:460] Webserver started at http://127.17.52.130:45043/ using document root <none> and password file <none>
I20260927 03:18:38.499202 27052 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.499469 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.499648 27052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.500835 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "52a6b66586f9433cad95aefc89c34c20"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "8fea00c3615e248ae0109bc6c370cf0b"
server_key_iv: "343c2a87fb7e859dfea2903b1a580463"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.501188 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "52a6b66586f9433cad95aefc89c34c20"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "8fea00c3615e248ae0109bc6c370cf0b"
server_key_iv: "343c2a87fb7e859dfea2903b1a580463"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.502481 27052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:38.503185 27066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.503343 27052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:38.503440 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "52a6b66586f9433cad95aefc89c34c20"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "8fea00c3615e248ae0109bc6c370cf0b"
server_key_iv: "343c2a87fb7e859dfea2903b1a580463"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.503518 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.513283 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.513563 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.513715 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.513912 27052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:38.514221 27052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:38.514276 27052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.514317 27052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:38.514345 27052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.519795 27052 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:35669
I20260927 03:18:38.519866 27179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:35669 every 8 connection(s)
I20260927 03:18:38.520200 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:38.525194 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.520985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:58381 (local address 127.17.52.190:43793)
0927 03:18:38.521028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.521030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.521254 (+ 224us) server_negotiation.cc:407] Connection header received
0927 03:18:38.521716 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.521717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.521727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.521782 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.522395 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.522897 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.523816 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.524020 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.524073 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.524910 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.524912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.524912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.524916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.524930 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.524934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.524985 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.525111 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.525135 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:38.525343 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.520983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43793 (local address 127.17.52.130:58381)
0927 03:18:38.521141 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:38.521171 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:38.521631 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:38.521836 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:38.521839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:38.521846 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:38.522329 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.522332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.522971 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.522972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.523742 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.523743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.524089 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.524089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.524241 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.524780 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:38.524780 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:38.524809 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:38.524831 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:38.524987 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:38.524990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:38.524992 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:18:38.525078 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:18:38.525153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260927 03:18:38.525867 27180 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43793
I20260927 03:18:38.525967 27180 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:38.526139 27180 heartbeater.cc:511] Master 127.17.52.190:43793 requested a full tablet report, sending...
I20260927 03:18:38.526520 26861 ts_manager.cc:194] Registered new tserver with Master: 52a6b66586f9433cad95aefc89c34c20 (127.17.52.130:35669)
I20260927 03:18:38.526896 26861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.130:58381
I20260927 03:18:38.527688 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27052
I20260927 03:18:38.527755 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:38.527944 17618 external_mini_cluster.cc:1428] Setting key a5c02ae94b740ea0ca3ab1ece95ae521
I20260927 03:18:38.528792 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--builtin_ntp_servers=127.17.52.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:18:38.608150 27183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.608353 27183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.608374 27183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.609553 27183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.609596 27183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.609620 27183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:38.610930 27183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.611169 27183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.611462 27183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:38.616431 27188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.616465 27191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.616449 27189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.616787 27183 server_base.cc:1034] running on GCE node
I20260927 03:18:38.617017 27183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.617270 27183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.618430 27183 hybrid_clock.cc:648] HybridClock initialized: now 1790479118618434 us; error 48 us; skew 500 ppm
I20260927 03:18:38.619675 27183 webserver.cc:460] Webserver started at http://127.17.52.131:45759/ using document root <none> and password file <none>
I20260927 03:18:38.619935 27183 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.620004 27183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.620163 27183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.621294 27183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "208514395694402eba2c6fc9cb488d13"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "63a2cfda9d49dab0d1ddfaa3515ec976"
server_key_iv: "0cf99ce65928ea9bdc5e31969a076354"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.621647 27183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "208514395694402eba2c6fc9cb488d13"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "63a2cfda9d49dab0d1ddfaa3515ec976"
server_key_iv: "0cf99ce65928ea9bdc5e31969a076354"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.622948 27183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:38.623703 27197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.623934 27183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:18:38.624030 27183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "208514395694402eba2c6fc9cb488d13"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "63a2cfda9d49dab0d1ddfaa3515ec976"
server_key_iv: "0cf99ce65928ea9bdc5e31969a076354"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.624112 27183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.640276 27183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.640668 27183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.640830 27183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.641053 27183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:38.641381 27183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:38.641434 27183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.641475 27183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:38.641501 27183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.647007 27183 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:40305
I20260927 03:18:38.647106 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:40305 every 8 connection(s)
I20260927 03:18:38.647413 27183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:38.652791 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.648149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:51997 (local address 127.17.52.190:43793)
0927 03:18:38.648198 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.648201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.648443 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:18:38.648999 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.649000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.649011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.649063 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.649700 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.650186 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.651259 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.651497 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.651569 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.652528 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.652529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.652529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.652533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.652543 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.652546 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.652606 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.652702 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.652721 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:18:38.653072 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.648150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43793 (local address 127.17.52.131:51997)
0927 03:18:38.648328 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:38.648362 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:38.648913 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:38.649118 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:38.649121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:38.649130 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:38.649633 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.649638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.650324 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.650326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.651176 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:38.651179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:38.651620 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:38.651622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:38.651757 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:38.652398 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:38.652398 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:38.652427 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:38.652451 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:38.652625 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:38.652629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:38.652632 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:38.652730 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:18:38.652818 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260927 03:18:38.653308 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27183
I20260927 03:18:38.653427 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:38.653635 27311 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43793
I20260927 03:18:38.653658 17618 external_mini_cluster.cc:1428] Setting key 4988e5f0b763f09afbf7d0897b74e35c
I20260927 03:18:38.653748 27311 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:38.653954 27311 heartbeater.cc:511] Master 127.17.52.190:43793 requested a full tablet report, sending...
I20260927 03:18:38.654367 26860 ts_manager.cc:194] Registered new tserver with Master: 208514395694402eba2c6fc9cb488d13 (127.17.52.131:40305)
I20260927 03:18:38.654824 26860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.131:51997
I20260927 03:18:38.654831 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:18:38.656940 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:38.656371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.17.52.190:43793)
0927 03:18:38.656412 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.656414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.656566 (+ 152us) server_negotiation.cc:407] Connection header received
0927 03:18:38.656677 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.656677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.656685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.656730 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.656875 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40866: BlockingRecv error: recv got EOF from 127.0.0.1:40866 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:38.657020 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26921
I20260927 03:18:38.662706 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27052
I20260927 03:18:38.667109 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27183
I20260927 03:18:38.671425 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 26830
2026-09-27T03:18:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (605 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T03:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:38Z Disabled control of system clock
I20260927 03:18:38.697820 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:46317
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:18:38.778781 27322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.778986 27322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.779011 27322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.780062 27322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:38.780095 27322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.780107 27322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.780119 27322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:38.780131 27322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:38.781488 27322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:46317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.781672 27322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.781893 27322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:38.786754 27327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.786803 27328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.786801 27330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.786944 27322 server_base.cc:1034] running on GCE node
I20260927 03:18:38.787242 27322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.787523 27322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.788632 27322 hybrid_clock.cc:648] HybridClock initialized: now 1790479118788604 us; error 39 us; skew 500 ppm
I20260927 03:18:38.790014 27322 webserver.cc:460] Webserver started at http://127.17.52.190:41449/ using document root <none> and password file <none>
I20260927 03:18:38.790267 27322 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.790336 27322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.790477 27322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.791531 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "70d5e51830df4b6187f5e83e6ed7a8ef"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f98e01dc4398593780a44c1841d83890"
server_key_iv: "1045d4dad3e4089f0d901ab872c3ec2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.791879 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "70d5e51830df4b6187f5e83e6ed7a8ef"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f98e01dc4398593780a44c1841d83890"
server_key_iv: "1045d4dad3e4089f0d901ab872c3ec2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.793186 27322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:38.793919 27336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.794100 27322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:38.794201 27322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "70d5e51830df4b6187f5e83e6ed7a8ef"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f98e01dc4398593780a44c1841d83890"
server_key_iv: "1045d4dad3e4089f0d901ab872c3ec2d"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.794275 27322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.797487 27322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.797731 27322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.797865 27322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.801430 27322 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:46317
I20260927 03:18:38.801457 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:46317 every 8 connection(s)
I20260927 03:18:38.801776 27322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:38.802655 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:38.804832 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef: Bootstrap starting.
I20260927 03:18:38.805377 27389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:38.805660 27389 log.cc:824] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:38.806308 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef: No bootstrap required, opened a new log
I20260927 03:18:38.807849 27389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } }
I20260927 03:18:38.807973 27389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:38.807996 27389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d5e51830df4b6187f5e83e6ed7a8ef, State: Initialized, Role: FOLLOWER
I20260927 03:18:38.808163 27389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } }
I20260927 03:18:38.808223 27389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:38.808243 27389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:38.808272 27389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:38.808863 27389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } }
I20260927 03:18:38.808967 27389 leader_election.cc:302] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d5e51830df4b6187f5e83e6ed7a8ef; no voters:
I20260927 03:18:38.809134 27389 leader_election.cc:288] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:38.809211 27392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:38.809404 27389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:38.809409 27392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [term 1 LEADER]: Becoming Leader. State: Replica: 70d5e51830df4b6187f5e83e6ed7a8ef, State: Running, Role: LEADER
I20260927 03:18:38.809540 27392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } }
I20260927 03:18:38.810037 27393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } } }
I20260927 03:18:38.810119 27393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:38.810436 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d5e51830df4b6187f5e83e6ed7a8ef. Latest consensus state: current_term: 1 leader_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d5e51830df4b6187f5e83e6ed7a8ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 46317 } } }
I20260927 03:18:38.810518 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:38.810477 27401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:38.810985 27401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:38.811810 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27322
I20260927 03:18:38.811913 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:38.812105 17618 external_mini_cluster.cc:1428] Setting key d3a42bf669b2731daa8e66326bf212ba
I20260927 03:18:38.812489 27401 catalog_manager.cc:1347] Generated new cluster ID: b88e5f0c5978431995c4a184c36113a1
I20260927 03:18:38.812536 27401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:38.814311 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.812730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.17.52.190:46317)
0927 03:18:38.812916 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.812924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.812956 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:18:38.813075 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.813082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.813126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.813267 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.813878 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.813882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.813882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.813886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.813927 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.813940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.814009 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.814078 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.814132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260927 03:18:38.842252 27401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:38.842440 27401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:38.850106 27401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d5e51830df4b6187f5e83e6ed7a8ef: Generated new TSK 0
I20260927 03:18:38.850322 27401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:18:38.852959 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--builtin_ntp_servers=127.17.52.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:18:38.932947 27413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:38.933149 27413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:38.933173 27413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:38.934384 27413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:38.934427 27413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:38.934454 27413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:38.935772 27413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:38.936012 27413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:38.936321 27413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:38.941054 27421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.941193 27413 server_base.cc:1034] running on GCE node
W20260927 03:18:38.941062 27418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:38.941062 27419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:38.941491 27413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:38.941707 27413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:38.942842 27413 hybrid_clock.cc:648] HybridClock initialized: now 1790479118942817 us; error 39 us; skew 500 ppm
I20260927 03:18:38.944016 27413 webserver.cc:460] Webserver started at http://127.17.52.129:40585/ using document root <none> and password file <none>
I20260927 03:18:38.944242 27413 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:38.944298 27413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:38.944473 27413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:38.945547 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "67b86a2cb18c404da29c12ff1766ab53"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f12618228e5a055ec29503e1d9de6fc4"
server_key_iv: "02dbf814bfb7da75b9f2c58c8f8fdf19"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.945906 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "67b86a2cb18c404da29c12ff1766ab53"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f12618228e5a055ec29503e1d9de6fc4"
server_key_iv: "02dbf814bfb7da75b9f2c58c8f8fdf19"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.947157 27413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:18:38.947816 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.947993 27413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:38.948068 27413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "67b86a2cb18c404da29c12ff1766ab53"
format_stamp: "Formatted at 2026-09-27 03:18:38 on dist-test-slave-7j43"
server_key: "f12618228e5a055ec29503e1d9de6fc4"
server_key_iv: "02dbf814bfb7da75b9f2c58c8f8fdf19"
server_key_version: "encryptionkey@0"
I20260927 03:18:38.948118 27413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:38.951260 27413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:38.951459 27413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:38.951546 27413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:38.951689 27413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:38.951913 27413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:38.951942 27413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.951961 27413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:38.951974 27413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:38.957409 27413 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:43507
I20260927 03:18:38.957475 27540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:43507 every 8 connection(s)
I20260927 03:18:38.957777 27413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:18:38.958127 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27413
I20260927 03:18:38.958209 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:38.958411 17618 external_mini_cluster.cc:1428] Setting key db0c3208a4702f74e8bf29cbf3f445ee
I20260927 03:18:38.959416 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--builtin_ntp_servers=127.17.52.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 03:18:38.961071 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.958518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:39013 (local address 127.17.52.190:46317)
0927 03:18:38.958582 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:38.958583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:38.958837 (+ 254us) server_negotiation.cc:407] Connection header received
0927 03:18:38.959405 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:38.959406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:38.959412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:38.959453 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:38.960772 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:38.960773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:38.960773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:38.960778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:38.960799 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:38.960805 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:38.960879 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:38.960967 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 03:18:38.960983 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:18:38.961227 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:38.958493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:46317 (local address 127.17.52.129:39013)
0927 03:18:38.958698 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:38.958739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:38.959297 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:38.959528 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:38.959533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:38.959542 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:38.960603 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:38.960603 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:38.960641 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:38.960675 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:38.960886 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:38.960893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:38.960896 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:38.960974 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:18:38.961033 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260927 03:18:38.961688 27541 heartbeater.cc:347] Connected to a master server at 127.17.52.190:46317
I20260927 03:18:38.961794 27541 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:38.961951 27541 heartbeater.cc:511] Master 127.17.52.190:46317 requested a full tablet report, sending...
I20260927 03:18:38.962416 27353 ts_manager.cc:194] Registered new tserver with Master: 67b86a2cb18c404da29c12ff1766ab53 (127.17.52.129:43507)
W20260927 03:18:39.038854 27544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:39.039053 27544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:39.039075 27544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:39.040295 27544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:39.040335 27544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:39.040360 27544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:39.041771 27544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:39.042018 27544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:39.042300 27544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:18:39.047211 27544 server_base.cc:1034] running on GCE node
W20260927 03:18:39.047186 27552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:39.047230 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:39.047197 27550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:39.047569 27544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:39.047847 27544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:39.048993 27544 hybrid_clock.cc:648] HybridClock initialized: now 1790479119048976 us; error 31 us; skew 500 ppm
I20260927 03:18:39.050185 27544 webserver.cc:460] Webserver started at http://127.17.52.130:34837/ using document root <none> and password file <none>
I20260927 03:18:39.050465 27544 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:39.050534 27544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:39.050666 27544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:39.051807 27544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "a1e9e62620ac4e84a3b248b8c53c8850"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "d2cf08c1ae4b47d1875abb1377b20c04"
server_key_iv: "8928b8d0ee149afd55499d9636bb42d7"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.052158 27544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "a1e9e62620ac4e84a3b248b8c53c8850"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "d2cf08c1ae4b47d1875abb1377b20c04"
server_key_iv: "8928b8d0ee149afd55499d9636bb42d7"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.053519 27544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:18:39.054266 27558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.054441 27544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:39.054538 27544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "a1e9e62620ac4e84a3b248b8c53c8850"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "d2cf08c1ae4b47d1875abb1377b20c04"
server_key_iv: "8928b8d0ee149afd55499d9636bb42d7"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.054621 27544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:39.057861 27544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:39.058105 27544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:39.058243 27544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:39.058445 27544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:39.058730 27544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:39.058785 27544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.058825 27544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:39.058851 27544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.064209 27544 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:38559
I20260927 03:18:39.064267 27671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:38559 every 8 connection(s)
I20260927 03:18:39.064643 27544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:39.067418 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:39.065392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:43203 (local address 127.17.52.190:46317)
0927 03:18:39.065482 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:39.065484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:39.065679 (+ 195us) server_negotiation.cc:407] Connection header received
0927 03:18:39.066114 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:39.066115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:39.066120 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:39.066182 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:39.067226 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:39.067227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:39.067228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:39.067231 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:39.067245 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:39.067249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:39.067296 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:39.067352 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:18:39.067363 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:18:39.067648 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:39.065354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:46317 (local address 127.17.52.130:43203)
0927 03:18:39.065556 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:39.065590 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:39.066031 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:39.066234 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:39.066237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:39.066245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:39.067086 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:39.067086 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:39.067114 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:39.067141 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:39.067304 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:39.067307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:39.067309 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:18:39.067359 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:18:39.067407 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260927 03:18:39.068172 27672 heartbeater.cc:347] Connected to a master server at 127.17.52.190:46317
I20260927 03:18:39.068250 27672 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:39.068522 27672 heartbeater.cc:511] Master 127.17.52.190:46317 requested a full tablet report, sending...
I20260927 03:18:39.068938 27353 ts_manager.cc:194] Registered new tserver with Master: a1e9e62620ac4e84a3b248b8c53c8850 (127.17.52.130:38559)
I20260927 03:18:39.074462 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27544
I20260927 03:18:39.074535 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:39.074721 17618 external_mini_cluster.cc:1428] Setting key f8e522eb84616dfbad7091395d98262e
I20260927 03:18:39.075580 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--builtin_ntp_servers=127.17.52.148:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:18:39.156894 27675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:39.157135 27675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:39.157171 27675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:39.158391 27675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:39.158450 27675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:39.158494 27675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:39.159817 27675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:39.160076 27675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:39.160358 27675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:39.165294 27681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:39.165270 27683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:39.165437 27675 server_base.cc:1034] running on GCE node
W20260927 03:18:39.165301 27680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:39.165735 27675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:39.165934 27675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:39.167073 27675 hybrid_clock.cc:648] HybridClock initialized: now 1790479119167050 us; error 34 us; skew 500 ppm
I20260927 03:18:39.168437 27675 webserver.cc:460] Webserver started at http://127.17.52.131:44167/ using document root <none> and password file <none>
I20260927 03:18:39.168718 27675 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:39.168771 27675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:39.168908 27675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:39.170027 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "07e2373d34284218b069b2e2c56b2485"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "6e38cdfa3e99fcdb33a7a79ed8e8585d"
server_key_iv: "2d2831beb60681f664ba5be18f256299"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.170377 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "07e2373d34284218b069b2e2c56b2485"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "6e38cdfa3e99fcdb33a7a79ed8e8585d"
server_key_iv: "2d2831beb60681f664ba5be18f256299"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.171664 27675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:18:39.172350 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.172519 27675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:39.172638 27675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "07e2373d34284218b069b2e2c56b2485"
format_stamp: "Formatted at 2026-09-27 03:18:39 on dist-test-slave-7j43"
server_key: "6e38cdfa3e99fcdb33a7a79ed8e8585d"
server_key_iv: "2d2831beb60681f664ba5be18f256299"
server_key_version: "encryptionkey@0"
I20260927 03:18:39.172699 27675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:39.175829 27675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:39.176017 27675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:39.176103 27675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:39.176242 27675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:39.176474 27675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:39.176503 27675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.176522 27675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:39.176534 27675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:39.181896 27675 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:36249
I20260927 03:18:39.181972 27802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:36249 every 8 connection(s)
I20260927 03:18:39.182253 27675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:18:39.184988 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:39.182956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:48353 (local address 127.17.52.190:46317)
0927 03:18:39.183016 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:39.183018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:39.183257 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:18:39.183665 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:39.183666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:39.183672 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:39.183715 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:39.184792 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:39.184793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:39.184793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:18:39.184798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:39.184811 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:18:39.184815 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:39.184867 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:39.184925 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:18:39.184937 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:18:39.185148 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:39.182949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:46317 (local address 127.17.52.131:48353)
0927 03:18:39.183133 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:39.183170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:39.183589 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:39.183770 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:39.183774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:39.183781 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:39.184623 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:18:39.184624 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:39.184686 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:18:39.184713 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:39.184866 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:39.184868 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:39.184871 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:39.184921 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:18:39.184965 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260927 03:18:39.185614 27803 heartbeater.cc:347] Connected to a master server at 127.17.52.190:46317
I20260927 03:18:39.185704 27803 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:39.185808 27803 heartbeater.cc:511] Master 127.17.52.190:46317 requested a full tablet report, sending...
I20260927 03:18:39.186131 27353 ts_manager.cc:194] Registered new tserver with Master: 07e2373d34284218b069b2e2c56b2485 (127.17.52.131:36249)
I20260927 03:18:39.190039 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27675
I20260927 03:18:39.190114 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:39.190294 17618 external_mini_cluster.cc:1428] Setting key 4412e7d014b3d6f1198d8db4f2c27277
I20260927 03:18:39.191098 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:18:39.192924 27412 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:18:39.192402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.17.52.190:46317)
0927 03:18:39.192459 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:39.192461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:39.192603 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:18:39.192693 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:39.192694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:39.192697 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:39.192743 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:39.192851 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50700: BlockingRecv error: recv got EOF from 127.0.0.1:50700 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:18:39.193053 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27413
I20260927 03:18:39.197773 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27544
I20260927 03:18:39.202587 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27675
I20260927 03:18:39.206571 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27322
2026-09-27T03:18:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (534 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:39 dist-test-slave-7j43 krb5kdc[27813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:39 dist-test-slave-7j43 krb5kdc[27813](info): set up 2 sockets
Sep 27 03:18:39 dist-test-slave-7j43 krb5kdc[27813](info): commencing operation
krb5kdc: starting...
W20260927 03:18:40.233958 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:18:40.347182 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44171
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:40.427008 27826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:40.427208 27826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:40.427229 27826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:40.428288 27826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:18:40.428321 27826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:40.428335 27826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:40.428347 27826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:18:40.428359 27826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:18:40.429759 27826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:40.429951 27826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:40.430114 27826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:40.434798 27832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.434782 27831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.434782 27834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:40.435056 27826 server_base.cc:1034] running on GCE node
I20260927 03:18:40.435351 27826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:40.435585 27826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:40.436714 27826 hybrid_clock.cc:648] HybridClock initialized: now 1790479120436699 us; error 24 us; skew 500 ppm
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:40.439128 27826 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:18:40.439580 27826 webserver.cc:460] Webserver started at http://127.17.52.190:44291/ using document root <none> and password file <none>
I20260927 03:18:40.439819 27826 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:40.439889 27826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:40.440037 27826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:40.441154 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "838ba28a7d6843d299537cdf57215f1b"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "10b53b3ef8da30ae0821d21d706d6a2b"
server_key_iv: "03a654e4a9393d6f92a8ae6b193b66f2"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.441514 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "838ba28a7d6843d299537cdf57215f1b"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "10b53b3ef8da30ae0821d21d706d6a2b"
server_key_iv: "03a654e4a9393d6f92a8ae6b193b66f2"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.442809 27826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:18:40.443497 27841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.443665 27826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:40.443770 27826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "838ba28a7d6843d299537cdf57215f1b"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "10b53b3ef8da30ae0821d21d706d6a2b"
server_key_iv: "03a654e4a9393d6f92a8ae6b193b66f2"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.443845 27826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:40.454005 27826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:40.455747 27826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:40.455907 27826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:40.459828 27826 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:44171
I20260927 03:18:40.459860 27893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:44171 every 8 connection(s)
I20260927 03:18:40.460210 27826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:18:40.461249 27894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:40.461974 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27826
I20260927 03:18:40.462052 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:18:40.462260 17618 external_mini_cluster.cc:1428] Setting key 3a9f1114d2f01a84220bf8375a474001
I20260927 03:18:40.464359 27894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: Bootstrap starting.
I20260927 03:18:40.465005 27894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:40.465312 27894 log.cc:824] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:40.465991 27894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: No bootstrap required, opened a new log
I20260927 03:18:40.467957 27894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } }
I20260927 03:18:40.468139 27894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:40.468173 27894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838ba28a7d6843d299537cdf57215f1b, State: Initialized, Role: FOLLOWER
I20260927 03:18:40.468307 27894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } }
I20260927 03:18:40.468405 27894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:18:40.468437 27894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:18:40.468483 27894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:40.469046 27894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } }
I20260927 03:18:40.469158 27894 leader_election.cc:302] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838ba28a7d6843d299537cdf57215f1b; no voters:
I20260927 03:18:40.469352 27894 leader_election.cc:288] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:18:40.469378 27899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:40.469515 27899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [term 1 LEADER]: Becoming Leader. State: Replica: 838ba28a7d6843d299537cdf57215f1b, State: Running, Role: LEADER
I20260927 03:18:40.469595 27899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } }
I20260927 03:18:40.469666 27894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:18:40.470054 27901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838ba28a7d6843d299537cdf57215f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } } }
I20260927 03:18:40.470227 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838ba28a7d6843d299537cdf57215f1b. Latest consensus state: current_term: 1 leader_uuid: "838ba28a7d6843d299537cdf57215f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ba28a7d6843d299537cdf57215f1b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44171 } } }
I20260927 03:18:40.470954 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [sys.catalog]: This master's current role is: LEADER
I20260927 03:18:40.470156 27901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b [sys.catalog]: This master's current role is: LEADER
W20260927 03:18:40.471318 27914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:18:40.471431 27914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:18:40.471518 27916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:18:40.471686 27916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:18:40.473302 27916 catalog_manager.cc:1347] Generated new cluster ID: c70da718a7ed401199e6e43c70495f59
I20260927 03:18:40.473302 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:40.462768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.17.52.190:44171)
0927 03:18:40.462944 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:40.462952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:40.462976 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:18:40.463382 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:40.463388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:40.463436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:40.463543 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:40.464329 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.465348 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.466101 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.466508 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.466638 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:40.469098 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:40.469102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:40.469103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:40.469122 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:40.471317 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.471750 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.471753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.471754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.471887 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.472106 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.472107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.472107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.472407 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:40.472535 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:40.472981 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 03:18:40.473111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260927 03:18:40.473373 27916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:18:40.481369 27916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:18:40.482095 27916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:18:40.495083 27916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: Generated new TSK 0
I20260927 03:18:40.495293 27916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:18:40.524847 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:40.604636 27920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:40.604831 27920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:40.604857 27920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:40.605923 27920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:40.605955 27920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:40.605980 27920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:18:40.607327 27920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:40.607506 27920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:40.607664 27920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:40.612699 27928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.612701 27925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.612742 27926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:40.612771 27920 server_base.cc:1034] running on GCE node
I20260927 03:18:40.613180 27920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:40.613363 27920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:40.614495 27920 hybrid_clock.cc:648] HybridClock initialized: now 1790479120614482 us; error 29 us; skew 500 ppm
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:40.616931 27920 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:18:40.617383 27920 webserver.cc:460] Webserver started at http://127.17.52.129:42497/ using document root <none> and password file <none>
I20260927 03:18:40.617599 27920 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:40.617666 27920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:40.617832 27920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:40.618831 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "3c034895f6624d2fb75c07508c7c3021"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "885c25eae57a7bc48d5cc308fb2ff6b4"
server_key_iv: "91730928f2319b31dcbfdc1bb826a979"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.619240 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "3c034895f6624d2fb75c07508c7c3021"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "885c25eae57a7bc48d5cc308fb2ff6b4"
server_key_iv: "91730928f2319b31dcbfdc1bb826a979"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.620492 27920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:18:40.621230 27935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.621417 27920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:18:40.621513 27920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "3c034895f6624d2fb75c07508c7c3021"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "885c25eae57a7bc48d5cc308fb2ff6b4"
server_key_iv: "91730928f2319b31dcbfdc1bb826a979"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.621596 27920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:40.641821 27920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:40.643481 27920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:40.643633 27920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:40.643863 27920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:40.644150 27920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:40.644202 27920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.644251 27920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:40.644282 27920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.650309 27920 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:34855
I20260927 03:18:40.650357 28048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:34855 every 8 connection(s)
I20260927 03:18:40.650638 27920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:40.658802 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:40.651284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47139 (local address 127.17.52.190:44171)
0927 03:18:40.651325 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:40.651327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:40.651515 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:18:40.651966 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:40.651967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:40.651976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:40.652031 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:40.652932 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.653424 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.654345 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.654542 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.654590 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:40.656504 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:40.656505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:40.656505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:40.656511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:40.657733 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.658068 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.658069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.658069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.658119 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.658324 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.658324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.658325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.658418 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:40.658468 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:40.658678 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 03:18:40.658739 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:18:40.658953 28051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:40.651246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44171 (local address 127.17.52.129:47139)
0927 03:18:40.651391 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:40.651429 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:40.651887 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:40.652079 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:40.652082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:40.652674 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:40.652857 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:40.652861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.653497 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.653498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:40.654271 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:40.654272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.654609 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.654609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:40.654700 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:40.655100 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:40.655107 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:40.656422 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:40.657800 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:40.657806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:40.657807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:40.658005 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:40.658174 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:40.658175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:40.658175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:40.658270 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:40.658471 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:40.658475 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:40.658575 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 03:18:40.658652 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:18:40.658767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 03:18:40.659504 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 27920
I20260927 03:18:40.659502 28049 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44171
I20260927 03:18:40.659592 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:18:40.659596 28049 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:40.659798 28049 heartbeater.cc:511] Master 127.17.52.190:44171 requested a full tablet report, sending...
I20260927 03:18:40.659803 17618 external_mini_cluster.cc:1428] Setting key a2760fc0cf5051eea776e922d105dc9e
I20260927 03:18:40.660422 27857 ts_manager.cc:194] Registered new tserver with Master: 3c034895f6624d2fb75c07508c7c3021 (127.17.52.129:34855)
I20260927 03:18:40.661356 27857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:47139
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:18:40.682652 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:40.762809 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:40.762991 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:40.763013 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:40.764135 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:40.764176 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:40.764230 28054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:18:40.765712 28054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:40.765949 28054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:40.766161 28054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:40.770820 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.770871 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.770831 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:40.771148 28054 server_base.cc:1034] running on GCE node
I20260927 03:18:40.771399 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:40.771684 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:40.772819 28054 hybrid_clock.cc:648] HybridClock initialized: now 1790479120772804 us; error 32 us; skew 500 ppm
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:40.775365 28054 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:18:40.775770 28054 webserver.cc:460] Webserver started at http://127.17.52.130:43679/ using document root <none> and password file <none>
I20260927 03:18:40.775986 28054 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:40.776028 28054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:40.776189 28054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:40.777266 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "d94684fce92642f19a0c05f6a66967cb"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "755b64e12debeff85d487e0d5df5f34b"
server_key_iv: "8db69e2ecbdf953209738210f88fed2f"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.777609 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "d94684fce92642f19a0c05f6a66967cb"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "755b64e12debeff85d487e0d5df5f34b"
server_key_iv: "8db69e2ecbdf953209738210f88fed2f"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.778832 28054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:18:40.779475 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.779682 28054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:40.779752 28054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "d94684fce92642f19a0c05f6a66967cb"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "755b64e12debeff85d487e0d5df5f34b"
server_key_iv: "8db69e2ecbdf953209738210f88fed2f"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.779850 28054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:40.824949 28054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:40.826993 28054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:40.827147 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:40.827411 28054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:40.827795 28054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:40.827839 28054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.827885 28054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:40.827914 28054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.833662 28054 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:44103
I20260927 03:18:40.833720 28182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:44103 every 8 connection(s)
I20260927 03:18:40.834023 28054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:18:40.837313 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 28054
I20260927 03:18:40.837389 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:18:40.837569 17618 external_mini_cluster.cc:1428] Setting key 5f714ecb07c1c5d27762542777dfd961
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:40.843468 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:40.834720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:37217 (local address 127.17.52.190:44171)
0927 03:18:40.834778 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:40.834780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:40.834966 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:18:40.835419 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:40.835420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:40.835429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:40.835484 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:40.836345 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.836951 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.838127 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.838409 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.838502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:40.840736 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:40.840737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:40.840738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:40.840745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:40.842083 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.842501 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.842502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.842502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.842558 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:40.842825 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:40.842826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:40.842827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:40.842941 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:40.843016 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:40.843298 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 03:18:40.843374 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:18:40.843664 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:40.834679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44171 (local address 127.17.52.130:37217)
0927 03:18:40.834840 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:40.834881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:40.835342 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:40.835536 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:40.835539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:40.836075 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:40.836251 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:40.836256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.837025 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.837026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:40.838056 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:40.838058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:40.838511 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:40.838512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:40.838663 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:40.839170 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:40.839185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:40.840634 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:40.842192 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:40.842198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:40.842199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:40.842423 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:40.842639 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:40.842640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:40.842640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:40.842756 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:40.843007 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:40.843013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:40.843151 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:18:40.843265 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:18:40.843423 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260927 03:18:40.844249 28183 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44171
I20260927 03:18:40.844324 28183 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:40.844507 28183 heartbeater.cc:511] Master 127.17.52.190:44171 requested a full tablet report, sending...
I20260927 03:18:40.844887 27857 ts_manager.cc:194] Registered new tserver with Master: d94684fce92642f19a0c05f6a66967cb (127.17.52.130:44103)
I20260927 03:18:40.845265 27857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:37217
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:18:40.861055 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:18:40.939308 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:18:40.939489 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:18:40.939512 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:18:40.940646 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:18:40.940714 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:18:40.940766 28188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:18:40.942215 28188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:18:40.942471 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:18:40.942698 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:18:40.947463 28193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.947479 28196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:18:40.947474 28194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:18:40.947750 28188 server_base.cc:1034] running on GCE node
I20260927 03:18:40.947942 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:18:40.948172 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:18:40.949355 28188 hybrid_clock.cc:648] HybridClock initialized: now 1790479120949331 us; error 34 us; skew 500 ppm
Sep 27 03:18:40 dist-test-slave-7j43 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:18:40.955368 28188 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:18:40.955813 28188 webserver.cc:460] Webserver started at http://127.17.52.131:37389/ using document root <none> and password file <none>
I20260927 03:18:40.956043 28188 fs_manager.cc:359] Metadata directory not provided
I20260927 03:18:40.956110 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:18:40.956255 28188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:18:40.957363 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "2b51c10fc3994efca5195813d01ca992"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "ba07bc2c0be251a7a29e7e6e8a96a9f2"
server_key_iv: "f0a9ba116b2fbc9aff334b838073a2e5"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.957870 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "2b51c10fc3994efca5195813d01ca992"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "ba07bc2c0be251a7a29e7e6e8a96a9f2"
server_key_iv: "f0a9ba116b2fbc9aff334b838073a2e5"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.959168 28188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:18:40.959872 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:40.960042 28188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:18:40.960142 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "2b51c10fc3994efca5195813d01ca992"
format_stamp: "Formatted at 2026-09-27 03:18:40 on dist-test-slave-7j43"
server_key: "ba07bc2c0be251a7a29e7e6e8a96a9f2"
server_key_iv: "f0a9ba116b2fbc9aff334b838073a2e5"
server_key_version: "encryptionkey@0"
I20260927 03:18:40.960210 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:18:40.997159 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:18:40.999480 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:18:40.999639 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:18:40.999878 28188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:18:41.000182 28188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:18:41.000236 28188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:41.000280 28188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:18:41.000305 28188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:18:41.006048 28188 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:43147
I20260927 03:18:41.006106 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:43147 every 8 connection(s)
I20260927 03:18:41.006444 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:18:41 dist-test-slave-7j43 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:18:41.015085 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.007132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:48919 (local address 127.17.52.190:44171)
0927 03:18:41.007182 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.007184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.007395 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:18:41.007900 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.007901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.007911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.007960 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:41.008854 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.009369 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.010321 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.010494 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.010541 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.012610 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:41.012612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:41.012614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:41.012626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:41.013993 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:41.014338 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:41.014339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:41.014339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:41.014385 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:41.014587 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:41.014587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:41.014587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:41.014684 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:41.014735 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.014956 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.015022 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:41.015235 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.007105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44171 (local address 127.17.52.131:48919)
0927 03:18:41.007263 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.007297 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.007810 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.008015 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.008019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.008590 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:18:41.008781 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.008786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.009445 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.009446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.010251 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.010253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.010560 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.010561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.010655 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.011098 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:18:41.011106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:18:41.012488 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:18:41.014060 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:41.014065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:41.014067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:41.014273 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:41.014441 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:18:41.014441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:18:41.014442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:18:41.014531 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:18:41.014739 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:18:41.014742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:18:41.014851 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.014929 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.015038 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260927 03:18:41.015502 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 28188
I20260927 03:18:41.015618 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:18:41.015712 28317 heartbeater.cc:347] Connected to a master server at 127.17.52.190:44171
I20260927 03:18:41.015807 28317 heartbeater.cc:464] Registering TS with master...
I20260927 03:18:41.015836 17618 external_mini_cluster.cc:1428] Setting key 902d960621c87b8d88b45444a0bc83d8
I20260927 03:18:41.015949 28317 heartbeater.cc:511] Master 127.17.52.190:44171 requested a full tablet report, sending...
I20260927 03:18:41.016335 27857 ts_manager.cc:194] Registered new tserver with Master: 2b51c10fc3994efca5195813d01ca992 (127.17.52.131:43147)
I20260927 03:18:41.016717 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:18:41.016877 27857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:48919
I20260927 03:18:41.023711 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.017965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.17.52.190:44171)
0927 03:18:41.018022 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.018024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.018081 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:18:41.018127 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.018128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.018135 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.018178 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:18:41.018645 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.019088 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.019810 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.019978 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.020025 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.021316 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:18:41.021317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:18:41.021317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:18:41.021323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:18:41.022706 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:41.023007 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:41.023008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:41.023008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:41.023053 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:18:41.023214 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:18:41.023214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:18:41.023215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:18:41.023337 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:18:41.023407 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.023495 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.023599 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:18:41.025273 27857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:18:41.026894 27857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:18:41.033370 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:43147 (local address 127.0.0.1:58470)
0927 03:18:41.029310 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.029362 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.029426 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.029797 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.029800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.029802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:41.029891 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.029893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.031306 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.031307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.032363 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.032365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.033067 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.033148 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033161 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.033236 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.033291 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260927 03:18:41.033464 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:34855 (local address 127.0.0.1:56432)
0927 03:18:41.029705 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.029713 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.029760 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.030037 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.030038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.030040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:41.030091 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.030093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.030982 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.030983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.032301 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.032303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.033214 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.033275 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033278 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.033328 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.033359 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":84,"threads_started":1}
I20260927 03:18:41.033471 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.17.52.129:34855)
0927 03:18:41.029553 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.029557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.029813 (+ 256us) server_negotiation.cc:407] Connection header received
0927 03:18:41.029858 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.029860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.029875 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.029957 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:41.030183 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.030871 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.032386 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.033127 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033214 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033267 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.033342 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.033381 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260927 03:18:41.033931 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44103 (local address 127.0.0.1:57278)
0927 03:18:41.029537 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.029548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.029616 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.030131 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.030131 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.030134 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:41.030188 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.030190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.031437 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.031438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.032633 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.032636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.033654 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.033739 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033743 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.033815 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.033856 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":17,"thread_start_us":83,"threads_started":1}
I20260927 03:18:41.033951 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.17.52.130:44103)
0927 03:18:41.029484 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.029488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.029619 (+ 131us) server_negotiation.cc:407] Connection header received
0927 03:18:41.029703 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.029705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.029941 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.030028 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:41.030281 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.031325 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.032744 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.033471 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033546 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033614 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.033831 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.033872 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260927 03:18:41.034127 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.029392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.17.52.131:43147)
0927 03:18:41.029552 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.029556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.029567 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:18:41.029603 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.029605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.029622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.029714 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:41.030016 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.031188 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.032424 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.032975 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.033056 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.033114 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.034007 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.034052 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260927 03:18:41.034204 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 59e46ad1fcf84020b79b37144a2050ad (DEFAULT_TABLE table=test-table [id=c51edbfe22c344e2b62509e5c0e08e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:41.034560 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e46ad1fcf84020b79b37144a2050ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:41.034765 28251 tablet_service.cc:1467] Processing CreateTablet for tablet 59e46ad1fcf84020b79b37144a2050ad (DEFAULT_TABLE table=test-table [id=c51edbfe22c344e2b62509e5c0e08e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:41.035072 28251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e46ad1fcf84020b79b37144a2050ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:41.037174 28334 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: Bootstrap starting.
I20260927 03:18:41.037642 28335 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Bootstrap starting.
I20260927 03:18:41.037763 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 59e46ad1fcf84020b79b37144a2050ad (DEFAULT_TABLE table=test-table [id=c51edbfe22c344e2b62509e5c0e08e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:18:41.037966 28334 tablet_bootstrap.cc:654] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:41.038074 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e46ad1fcf84020b79b37144a2050ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:18:41.038271 28335 tablet_bootstrap.cc:654] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:41.038496 28334 log.cc:824] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:41.038672 28335 log.cc:824] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:41.039286 28334 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: No bootstrap required, opened a new log
I20260927 03:18:41.039348 28334 ts_tablet_manager.cc:1397] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:41.039458 28335 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: No bootstrap required, opened a new log
I20260927 03:18:41.039510 28335 ts_tablet_manager.cc:1397] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:18:41.040989 28338 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Bootstrap starting.
I20260927 03:18:41.041507 28338 tablet_bootstrap.cc:654] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Neither blocks nor log segments found. Creating new log.
I20260927 03:18:41.041428 28334 raft_consensus.cc:348] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.041455 28335 raft_consensus.cc:348] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.041604 28335 raft_consensus.cc:374] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:41.041604 28334 raft_consensus.cc:374] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:41.041643 28334 raft_consensus.cc:729] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c034895f6624d2fb75c07508c7c3021, State: Initialized, Role: FOLLOWER
I20260927 03:18:41.041643 28335 raft_consensus.cc:729] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b51c10fc3994efca5195813d01ca992, State: Initialized, Role: FOLLOWER
I20260927 03:18:41.041765 28335 consensus_queue.cc:260] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.041764 28334 consensus_queue.cc:260] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.042043 28335 ts_tablet_manager.cc:1428] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:18:41.042043 28334 ts_tablet_manager.cc:1428] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:18:41.042078 28049 heartbeater.cc:503] Master 127.17.52.190:44171 was elected leader, sending a full tablet report...
I20260927 03:18:41.042071 28338 log.cc:824] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:18:41.042181 28317 heartbeater.cc:503] Master 127.17.52.190:44171 was elected leader, sending a full tablet report...
I20260927 03:18:41.042861 28338 tablet_bootstrap.cc:492] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: No bootstrap required, opened a new log
I20260927 03:18:41.042932 28338 ts_tablet_manager.cc:1397] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:18:41.044384 28338 raft_consensus.cc:348] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.044495 28338 raft_consensus.cc:374] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:18:41.044518 28338 raft_consensus.cc:729] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94684fce92642f19a0c05f6a66967cb, State: Initialized, Role: FOLLOWER
I20260927 03:18:41.044667 28338 consensus_queue.cc:260] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.044951 28338 ts_tablet_manager.cc:1428] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:18:41.044986 28183 heartbeater.cc:503] Master 127.17.52.190:44171 was elected leader, sending a full tablet report...
I20260927 03:18:41.061579 28340 raft_consensus.cc:482] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:18:41.061671 28340 raft_consensus.cc:504] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.061990 28340 leader_election.cc:288] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d94684fce92642f19a0c05f6a66967cb (127.17.52.130:44103), 3c034895f6624d2fb75c07508c7c3021 (127.17.52.129:34855)
I20260927 03:18:41.065078 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.062176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:55869 (local address 127.17.52.130:44103)
0927 03:18:41.062221 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.062223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.062233 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:18:41.062302 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.062303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.062315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.062375 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:41.062545 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.063071 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064168 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.064748 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064800 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.064808 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.064977 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.065010 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:18:41.065068 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.062122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44103 (local address 127.17.52.131:55869)
0927 03:18:41.062177 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.062189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.062235 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.062394 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.062395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.062396 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:41.062475 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.062477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.063153 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.063154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.064081 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.064083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.064819 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.064899 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.064903 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.064961 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.065005 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":84}
I20260927 03:18:41.065316 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.062147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:34855 (local address 127.17.52.131:42541)
0927 03:18:41.062300 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.062310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:18:41.062364 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:18:41.062536 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:18:41.062537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:18:41.062539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:18:41.062598 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.062599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.063211 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.063211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.064091 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:18:41.064093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.064983 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:18:41.065060 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.065064 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:18:41.065122 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:18:41.065224 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260927 03:18:41.065383 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e46ad1fcf84020b79b37144a2050ad" candidate_uuid: "2b51c10fc3994efca5195813d01ca992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94684fce92642f19a0c05f6a66967cb" is_pre_election: true
I20260927 03:18:41.065533 28137 raft_consensus.cc:2454] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b51c10fc3994efca5195813d01ca992 in term 0.
I20260927 03:18:41.065734 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.062251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:42541 (local address 127.17.52.129:34855)
0927 03:18:41.062289 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.062291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.062374 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:18:41.062439 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.062440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.062450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.062507 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:18:41.062662 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.063097 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064212 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.064897 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.064970 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.064977 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.065628 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.065672 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:18:41.065749 28206 leader_election.cc:302] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b51c10fc3994efca5195813d01ca992, d94684fce92642f19a0c05f6a66967cb; no voters:
I20260927 03:18:41.065862 28340 raft_consensus.cc:2781] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:18:41.065930 28340 raft_consensus.cc:482] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:18:41.065960 28340 raft_consensus.cc:3037] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:41.066593 28340 raft_consensus.cc:504] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.066740 28340 leader_election.cc:288] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [CANDIDATE]: Term 1 election: Requested vote from peers d94684fce92642f19a0c05f6a66967cb (127.17.52.130:44103), 3c034895f6624d2fb75c07508c7c3021 (127.17.52.129:34855)
I20260927 03:18:41.066867 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e46ad1fcf84020b79b37144a2050ad" candidate_uuid: "2b51c10fc3994efca5195813d01ca992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94684fce92642f19a0c05f6a66967cb"
I20260927 03:18:41.066946 28137 raft_consensus.cc:3037] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:41.066989 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e46ad1fcf84020b79b37144a2050ad" candidate_uuid: "2b51c10fc3994efca5195813d01ca992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c034895f6624d2fb75c07508c7c3021"
I20260927 03:18:41.067117 28002 raft_consensus.cc:3037] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:18:41.067811 28137 raft_consensus.cc:2454] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b51c10fc3994efca5195813d01ca992 in term 1.
I20260927 03:18:41.067971 28206 leader_election.cc:302] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b51c10fc3994efca5195813d01ca992, d94684fce92642f19a0c05f6a66967cb; no voters:
I20260927 03:18:41.067943 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e46ad1fcf84020b79b37144a2050ad" candidate_uuid: "2b51c10fc3994efca5195813d01ca992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c034895f6624d2fb75c07508c7c3021" is_pre_election: true
I20260927 03:18:41.068027 28002 raft_consensus.cc:2454] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b51c10fc3994efca5195813d01ca992 in term 1.
I20260927 03:18:41.068053 28340 raft_consensus.cc:2781] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:18:41.068197 28340 raft_consensus.cc:686] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 1 LEADER]: Becoming Leader. State: Replica: 2b51c10fc3994efca5195813d01ca992, State: Running, Role: LEADER
I20260927 03:18:41.068290 28340 consensus_queue.cc:237] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } }
I20260927 03:18:41.069129 27858 catalog_manager.cc:5495] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b51c10fc3994efca5195813d01ca992 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "2b51c10fc3994efca5195813d01ca992" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b51c10fc3994efca5195813d01ca992" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 43147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 } health_report { overall_health: UNKNOWN } } }
I20260927 03:18:41.078033 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.075885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.17.52.131:43147)
0927 03:18:41.075929 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:18:41.075930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:18:41.075937 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:18:41.076002 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:18:41.076003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:18:41.076016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:18:41.076069 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:18:41.076170 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.076660 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.077371 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:18:41.077544 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:18:41.077615 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:18:41.077729 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:18:41.077817 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:18:41.077874 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:18:41.077907 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:18:41.077921 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:18:41.079545 28002 raft_consensus.cc:1261] T 59e46ad1fcf84020b79b37144a2050ad P 3c034895f6624d2fb75c07508c7c3021 [term 1 FOLLOWER]: Refusing update from remote peer 2b51c10fc3994efca5195813d01ca992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:41.079569 28137 raft_consensus.cc:1261] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 1 FOLLOWER]: Refusing update from remote peer 2b51c10fc3994efca5195813d01ca992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:18:41.079806 28340 consensus_queue.cc:1035] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c034895f6624d2fb75c07508c7c3021" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:41.079928 28344 consensus_queue.cc:1035] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d94684fce92642f19a0c05f6a66967cb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:18:41.083220 28349 mvcc.cc:205] Tried to move back new op lower bound from 7333802479939596288 to 7333802479896297472. Current Snapshot: MvccSnapshot[applied={T|T < 7333802479939596288 or (T in {7333802479939596288})}]
I20260927 03:18:41.083527 28348 mvcc.cc:205] Tried to move back new op lower bound from 7333802479939596288 to 7333802479896297472. Current Snapshot: MvccSnapshot[applied={T|T < 7333802479939596288 or (T in {7333802479939596288})}]
I20260927 03:18:41.084560 27858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:41.084702 27858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:18:41.085439 27858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 838ba28a7d6843d299537cdf57215f1b: Sending DeleteTablet for 3 replicas of tablet 59e46ad1fcf84020b79b37144a2050ad
I20260927 03:18:41.085753 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e46ad1fcf84020b79b37144a2050ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:41 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56432
I20260927 03:18:41.085780 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e46ad1fcf84020b79b37144a2050ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:41 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:57278
I20260927 03:18:41.086014 28357 tablet_replica.cc:330] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: stopping tablet replica
I20260927 03:18:41.086000 28251 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e46ad1fcf84020b79b37144a2050ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:18:41 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:58470
I20260927 03:18:41.086143 28357 raft_consensus.cc:2229] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:18:41.086256 28357 raft_consensus.cc:2258] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:41.086499 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27920
I20260927 03:18:41.086714 28357 ts_tablet_manager.cc:1905] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:18:41.088966 28357 ts_tablet_manager.cc:1918] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:41.089097 28357 log.cc:1196] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/59e46ad1fcf84020b79b37144a2050ad
I20260927 03:18:41.089468 28357 ts_tablet_manager.cc:1939] T 59e46ad1fcf84020b79b37144a2050ad P d94684fce92642f19a0c05f6a66967cb: Deleting consensus metadata
I20260927 03:18:41.089659 28359 tablet_replica.cc:330] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: stopping tablet replica
I20260927 03:18:41.089774 28359 raft_consensus.cc:2229] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:18:41.089860 28359 raft_consensus.cc:2258] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:18:41.089892 27844 catalog_manager.cc:4841] TS d94684fce92642f19a0c05f6a66967cb (127.17.52.130:44103): tablet 59e46ad1fcf84020b79b37144a2050ad (table test-table [id=c51edbfe22c344e2b62509e5c0e08e53]) successfully deleted
I20260927 03:18:41.092980 28359 ts_tablet_manager.cc:1905] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:18:41.093927 27843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:34855 (error 108)
I20260927 03:18:41.094339 28359 ts_tablet_manager.cc:1918] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:18:41.094421 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:18:41.094298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:34855 (local address 127.0.0.1:56434)
0927 03:18:41.094347 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:18:41.094385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:18:41.094460 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 28054
W20260927 03:18:41.094529 27843 catalog_manager.cc:4584] TS 3c034895f6624d2fb75c07508c7c3021 (127.17.52.129:34855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59e46ad1fcf84020b79b37144a2050ad: Network error: Client connection negotiation failed: client connection to 127.17.52.129:34855: connect: Connection refused (error 111)
I20260927 03:18:41.095149 28359 log.cc:1196] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/59e46ad1fcf84020b79b37144a2050ad
I20260927 03:18:41.095368 28359 ts_tablet_manager.cc:1939] T 59e46ad1fcf84020b79b37144a2050ad P 2b51c10fc3994efca5195813d01ca992: Deleting consensus metadata
I20260927 03:18:41.095679 27842 catalog_manager.cc:4841] TS 2b51c10fc3994efca5195813d01ca992 (127.17.52.131:43147): tablet 59e46ad1fcf84020b79b37144a2050ad (table test-table [id=c51edbfe22c344e2b62509e5c0e08e53]) successfully deleted
I20260927 03:18:41.098834 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 28188
I20260927 03:18:41.103794 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 27826
2026-09-27T03:18:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1898 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:18:41 dist-test-slave-7j43 krb5kdc[28363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:18:41 dist-test-slave-7j43 krb5kdc[28363](info): set up 2 sockets
Sep 27 03:18:41 dist-test-slave-7j43 krb5kdc[28363](info): commencing operation
krb5kdc: starting...
W20260927 03:18:42.146023 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:18:42 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:18:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangeradmin_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangeradmin_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangerlookup_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangerlookup_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
I20260927 03:18:42.297534 17618 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRLekJh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/postgres -l logfile start
2026-09-27 03:18:44.636 UTC [28393] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:18:44.637 UTC [28393] LOG: listening on IPv4 address "127.17.52.148", port 51641
2026-09-27 03:18:44.640 UTC [28393] LOG: listening on Unix socket "/tmp/.s.PGSQL.51641"
2026-09-27 03:18:44.674 UTC [28396] LOG: database system was shut down at 2026-09-27 03:18:43 UTC
2026-09-27 03:18:44.678 UTC [28393] LOG: database system is ready to accept connections
I20260927 03:18:45.637918 17618 mini_postgres.cc:96] Postgres bound to 51641
2026-09-27 03:18:45.644 UTC [28404] FATAL: database "slave" does not exist
127.17.52.148:51641 - accepting connections
I20260927 03:18:45.645066 17618 mini_ranger.cc:161] Starting Ranger...
I20260927 03:18:45.659561 17618 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:18:46.082249 17618 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:18:46.082329 17618 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin
2026-09-27 03:18:46,381 [I] DB FLAVOR :POSTGRES
2026-09-27 03:18:46,381 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:18:46,381 [I] Checking connection..
2026-09-27 03:18:46,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:18:46,621 [I] Checking connection passed.
2026-09-27 03:18:46,621 [I] --------- Verifying version history table ---------
2026-09-27 03:18:46,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:18:46,862 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:18:46,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:18:47,108 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:18:47,109 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:18:47,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:18:47.355 UTC [28468] WARNING: there is no transaction in progress
2026-09-27 03:18:47,361 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:18:47,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:18:47,600 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:18:47,600 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:18:47,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:18:47,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:18:48,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7j43.c.gcp-upstream.internal','N') ;"
2026-09-27 03:18:48,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:18:48,530 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:18:48,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:18:48,762 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:18:48,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:18:48,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:18:49,237 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:18:49,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:18:50.110 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.125 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.139 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.152 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.167 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.183 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.237 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.243 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.257 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.267 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.277 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.289 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.299 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.305 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.312 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50.318 UTC [28576] WARNING: there is no transaction in progress
2026-09-27 03:18:50,902 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:18:50,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:18:51,139 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:18:51,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:18:51,374 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:18:51,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:18:51,619 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:18:51,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:18:51,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:18:52,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7j43.c.gcp-upstream.internal';"
2026-09-27 03:18:52,356 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:18:52,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:18:52,589 [I] DB_PATCHES have already been applied
I20260927 03:18:52.595485 17618 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRLekJh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:18:52.595546 17618 mini_ranger.cc:193] Using host: 127.17.52.148
I20260927 03:18:52.598490 17618 mini_ranger.cc:227] Ranger admin URL: http://127.17.52.148:53757
Sep 27, 2026 3:18:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:18:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:18:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:18:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:18:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:18:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:18:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:18:53.704552 17618 mini_ranger.cc:160] Time spent starting Ranger: real 8.059s user 0.000s sys 0.008s
Sep 27, 2026 3:19:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:19:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 3:19:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:19:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:19:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:19:15.914413 17618 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangerkms_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/rangerkms_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.52.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.52.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:19:15 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479155, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:19:15.997430 17618 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:19:16.006270 17618 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:19:16.743614 17618 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:19:16.743697 17618 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms
I20260927 03:19:16.743719 17618 mini_ranger_kms.cc:227] Using postgres at 127.17.52.148:51641
2026-09-27 03:19:17,110 [I] DB FLAVOR :POSTGRES
2026-09-27 03:19:17,110 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:19:17,110 [I] Checking connection
2026-09-27 03:19:17,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:19:17,339 [I] connection success
2026-09-27 03:19:17,339 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:19:17,339 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:19:17,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:19:17,560 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:19:17,560 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:19:17,560 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:19:17,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:51641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:19:17,820 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:19:18.291159 17618 mini_ranger_kms.cc:310] Created kms service
I20260927 03:19:18.469525 17618 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:19:18.511458 17618 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:19:18 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479155, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:18.891657 17618 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:19:18.891956 17618 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:19:18.891993 17618 mini_ranger_kms.cc:240] Using host: 127.17.52.148
I20260927 03:19:18.895539 17618 mini_ranger_kms.cc:276] Ranger KMS PID: 28756
I20260927 03:19:18.895677 17618 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.52.148:35465
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:19:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:19:21.128064 17618 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.131s user 0.009s sys 0.235s
I20260927 03:19:21.128176 17618 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:19:21.128288 17618 mini_ranger_kms.cc:401] 127.17.52.148:35465/kms/v1/keys
Sep 27 03:19:21 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:19:21 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
Sep 27 03:19:22 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
Sep 27 03:19:24 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:19:24.287328 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:37041
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:37041
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:35465/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:19:24.380950 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:19:24.381145 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:19:24.381167 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:19:24.382365 28807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:19:24.382405 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:19:24.382419 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:19:24.382431 28807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:19:24.382443 28807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:19:24.383963 28807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.52.148:35465/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:37041
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:19:24.384222 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:19:24.384461 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:19:24.389460 28807 server_base.cc:1034] running on GCE node
W20260927 03:19:24.389477 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:24.389500 28813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:24.389529 28815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:19:24.389904 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:19:24.390215 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:19:24.391351 28807 hybrid_clock.cc:648] HybridClock initialized: now 1790479164391331 us; error 38 us; skew 500 ppm
Sep 27 03:19:24 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479164, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:19:24.393711 28807 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:19:24.394079 28807 webserver.cc:460] Webserver started at http://127.17.52.190:45057/ using document root <none> and password file <none>
I20260927 03:19:24.394243 28807 fs_manager.cc:359] Metadata directory not provided
I20260927 03:19:24.394276 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:19:24.394374 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:19:24 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479164, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:24.445070 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "c8b55d7e04cd4dae8d21b871f3e672c8"
format_stamp: "Formatted at 2026-09-27 03:19:24 on dist-test-slave-7j43"
server_key: "b8b33ec1cbf301679f05db82a291d468"
server_key_iv: "0baffe2b947fc57a09f609a471ace25e"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:24.445479 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "c8b55d7e04cd4dae8d21b871f3e672c8"
format_stamp: "Formatted at 2026-09-27 03:19:24 on dist-test-slave-7j43"
server_key: "b8b33ec1cbf301679f05db82a291d468"
server_key_iv: "0baffe2b947fc57a09f609a471ace25e"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:24.447121 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:19:24.458096 28822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:24.458292 28807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:19:24.458415 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "c8b55d7e04cd4dae8d21b871f3e672c8"
format_stamp: "Formatted at 2026-09-27 03:19:24 on dist-test-slave-7j43"
server_key: "b8b33ec1cbf301679f05db82a291d468"
server_key_iv: "0baffe2b947fc57a09f609a471ace25e"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:24.458516 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:19:24.477669 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:19:24.479576 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:19:24.479733 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:19:24.484436 28807 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:37041
I20260927 03:19:24.484457 28874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:37041 every 8 connection(s)
I20260927 03:19:24.484861 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:19:24.485540 28875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:19:24.487690 28875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: Bootstrap starting.
I20260927 03:19:24.488269 28875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: Neither blocks nor log segments found. Creating new log.
I20260927 03:19:24.488610 28875 log.cc:824] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:19:24.489342 28875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: No bootstrap required, opened a new log
I20260927 03:19:24.491310 28875 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } }
I20260927 03:19:24.491446 28875 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:19:24.491468 28875 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b55d7e04cd4dae8d21b871f3e672c8, State: Initialized, Role: FOLLOWER
I20260927 03:19:24.491614 28875 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } }
I20260927 03:19:24.491685 28875 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:19:24.491707 28875 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:19:24.491772 28875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:19:24.492272 28875 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } }
I20260927 03:19:24.492362 28875 leader_election.cc:302] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b55d7e04cd4dae8d21b871f3e672c8; no voters:
I20260927 03:19:24.492550 28875 leader_election.cc:288] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:19:24.492595 28878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:19:24.492754 28878 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [term 1 LEADER]: Becoming Leader. State: Replica: c8b55d7e04cd4dae8d21b871f3e672c8, State: Running, Role: LEADER
I20260927 03:19:24.492826 28875 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:19:24.492908 28878 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } }
I20260927 03:19:24.493232 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 28807
I20260927 03:19:24.493259 28878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b55d7e04cd4dae8d21b871f3e672c8. Latest consensus state: current_term: 1 leader_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } } }
I20260927 03:19:24.493317 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:19:24.493335 28878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:19:24.493497 17618 external_mini_cluster.cc:1428] Setting key 929914ebe1d92b4db52ff1a888bbfe42
I20260927 03:19:24.494024 28880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b55d7e04cd4dae8d21b871f3e672c8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 37041 } } }
I20260927 03:19:24.494158 28880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:19:24.494418 28875 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:19:24.497192 28884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:19:24.498183 28884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:19:24 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:19:24.505074 28884 catalog_manager.cc:1347] Generated new cluster ID: c0cccf56d35f4d73aa856328829c8e7c
I20260927 03:19:24.505121 28884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:19:24.509426 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:24.494180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.17.52.190:37041)
0927 03:19:24.497028 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:24.497062 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:24.497131 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:19:24.497590 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:24.497602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:24.497694 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:24.497879 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:19:24.498928 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:24.500496 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:24.501686 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:24.502292 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:24.502397 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:24.505403 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:19:24.505406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:19:24.505407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:19:24.505418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:19:24.507377 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:24.507790 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:24.507794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:24.507794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:24.507919 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:24.508143 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:24.508144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:24.508144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:24.508434 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:19:24.508544 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:24.508862 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 03:19:24.508986 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2715,"thread_start_us":64,"threads_started":1}
I20260927 03:19:24.524078 28884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:19:24.524890 28884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:19:24.533018 28884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: Generated new TSK 0
I20260927 03:19:24.533258 28884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:19:25.224737 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:35465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:19:25.324065 28920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:19:25.324329 28920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:19:25.324363 28920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:19:25.326049 28920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:19:25.326107 28920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:19:25.326146 28920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:19:25.328197 28920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.52.148:35465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:19:25.328423 28920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:19:25.328717 28920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:19:25.335191 28928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.335597 28925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.335311 28926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:19:25.335376 28920 server_base.cc:1034] running on GCE node
I20260927 03:19:25.336479 28920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:19:25.336755 28920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:19:25.337940 28920 hybrid_clock.cc:648] HybridClock initialized: now 1790479165337912 us; error 43 us; skew 500 ppm
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:19:25.341221 28920 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:19:25.341701 28920 webserver.cc:460] Webserver started at http://127.17.52.129:43397/ using document root <none> and password file <none>
I20260927 03:19:25.341943 28920 fs_manager.cc:359] Metadata directory not provided
I20260927 03:19:25.341992 28920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:19:25.342109 28920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:25.359275 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "ccf8b6b4b778ea1839846a64b3308ba0"
server_key_iv: "48f41eda933ee3371100734a404d9fce"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.359794 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "ccf8b6b4b778ea1839846a64b3308ba0"
server_key_iv: "48f41eda933ee3371100734a404d9fce"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.365152 28920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260927 03:19:25.378042 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.378281 28920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:19:25.378376 28920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "ccf8b6b4b778ea1839846a64b3308ba0"
server_key_iv: "48f41eda933ee3371100734a404d9fce"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.378463 28920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:19:25.401132 28920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:19:25.403352 28920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:19:25.403576 28920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:19:25.403873 28920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:19:25.404273 28920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:19:25.404361 28920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.404459 28920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:19:25.404531 28920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.411827 28920 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46753
I20260927 03:19:25.412206 28920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:19:25.413105 29050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46753 every 8 connection(s)
I20260927 03:19:25.416816 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 28920
I20260927 03:19:25.416909 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:19:25.417110 17618 external_mini_cluster.cc:1428] Setting key e6d29c9e9d52c03213ae404e991aa18a
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:19:25.435801 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.413800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50089 (local address 127.17.52.190:37041)
0927 03:19:25.420709 (+ 6909us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.420711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.420822 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:19:25.422181 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.422182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.422196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.422243 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:19:25.423134 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.423928 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.427597 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.427919 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.427965 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.432487 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:19:25.432489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:19:25.432489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:19:25.432496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:19:25.434244 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.434723 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.434724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.434725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.434783 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.435043 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.435044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.435044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.435175 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:19:25.435227 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.435614 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.435698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6876,"thread_start_us":81,"threads_started":1}
I20260927 03:19:25.436605 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.413778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:37041 (local address 127.17.52.129:50089)
0927 03:19:25.420661 (+ 6883us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.420719 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.422046 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.422471 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.422475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.422739 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:19:25.423009 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.423016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.424042 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.424043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.425352 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.425355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.428056 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.428057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.428196 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.430739 (+ 2543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:19:25.430750 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:19:25.432377 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:19:25.434348 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.434354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.434357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.434635 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.434850 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.434851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.434851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.434974 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.435260 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:19:25.435265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:19:25.435453 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.436178 (+ 725us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.436353 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6820,"thread_start_us":78,"threads_started":1}
I20260927 03:19:25.437278 29051 heartbeater.cc:347] Connected to a master server at 127.17.52.190:37041
I20260927 03:19:25.437381 29051 heartbeater.cc:464] Registering TS with master...
I20260927 03:19:25.437569 29051 heartbeater.cc:511] Master 127.17.52.190:37041 requested a full tablet report, sending...
I20260927 03:19:25.438206 28839 ts_manager.cc:194] Registered new tserver with Master: 0300fdeb8edb4ad3ba4ffea758e7efa7 (127.17.52.129:46753)
I20260927 03:19:25.439075 28839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:50089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:19:25.457805 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:35465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:19:25.555472 29059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:19:25.555823 29059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:19:25.555912 29059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:19:25.557687 29059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:19:25.557790 29059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:19:25.557878 29059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:19:25.559903 29059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.52.148:35465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:19:25.560388 29059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:19:25.560734 29059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:19:25.566639 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.567301 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.567524 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:19:25.569618 29059 server_base.cc:1034] running on GCE node
I20260927 03:19:25.569898 29059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:19:25.570182 29059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:19:25.571352 29059 hybrid_clock.cc:648] HybridClock initialized: now 1790479165571331 us; error 35 us; skew 500 ppm
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:19:25.574424 29059 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:19:25.574934 29059 webserver.cc:460] Webserver started at http://127.17.52.130:37021/ using document root <none> and password file <none>
I20260927 03:19:25.575186 29059 fs_manager.cc:359] Metadata directory not provided
I20260927 03:19:25.575254 29059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:19:25.575400 29059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:25.592711 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "bca70d4c5310482ba1379b5617714a67"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "f7fe573da6c1051637d43dc3117e1146"
server_key_iv: "f31beb2683a288e279d278b210db67c1"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.593456 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "bca70d4c5310482ba1379b5617714a67"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "f7fe573da6c1051637d43dc3117e1146"
server_key_iv: "f31beb2683a288e279d278b210db67c1"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.595408 29059 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 03:19:25.608711 29074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.608888 29059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260927 03:19:25.608995 29059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "bca70d4c5310482ba1379b5617714a67"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "f7fe573da6c1051637d43dc3117e1146"
server_key_iv: "f31beb2683a288e279d278b210db67c1"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.609102 29059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:19:25.632602 29059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:19:25.634960 29059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:19:25.635169 29059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:19:25.635412 29059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:19:25.635811 29059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:19:25.635895 29059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.635967 29059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:19:25.636013 29059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.642215 29059 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:33013
I20260927 03:19:25.642660 29059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:19:25.644778 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 29059
I20260927 03:19:25.644881 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:19:25.645077 17618 external_mini_cluster.cc:1428] Setting key ddd47d178ceb2f3c1dfe17e93b543b6c
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
I20260927 03:19:25.661237 29188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:33013 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:19:25.675323 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:35465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:19:25.683871 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.661444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:47051 (local address 127.17.52.190:37041)
0927 03:19:25.661496 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.661499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.667142 (+ 5643us) server_negotiation.cc:407] Connection header received
0927 03:19:25.668017 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.668018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.668037 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.668068 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:19:25.669524 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.670342 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.672412 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.672822 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.672872 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.679613 (+ 6741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:19:25.679615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:19:25.679615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:19:25.679624 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:19:25.681397 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.682457 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.682458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.682459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.682528 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.682935 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.682936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.682936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.683107 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:19:25.683168 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.683671 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.683766 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:19:25.684409 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.661455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:37041 (local address 127.17.52.130:47051)
0927 03:19:25.666953 (+ 5498us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.667013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.667726 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.668281 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.668286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.668679 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:19:25.669409 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.669417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.670525 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.670526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.672312 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.672315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.673070 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.673071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.673229 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.673899 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:19:25.673916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:19:25.678015 (+ 4099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:19:25.681507 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.681514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.681517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.681813 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.682680 (+ 867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.682681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.682681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.682842 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.683226 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:19:25.683233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:19:25.683479 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.683975 (+ 496us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.684159 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"thread_start_us":105,"threads_started":1}
I20260927 03:19:25.685043 29189 heartbeater.cc:347] Connected to a master server at 127.17.52.190:37041
I20260927 03:19:25.685137 29189 heartbeater.cc:464] Registering TS with master...
I20260927 03:19:25.685333 29189 heartbeater.cc:511] Master 127.17.52.190:37041 requested a full tablet report, sending...
I20260927 03:19:25.685750 28839 ts_manager.cc:194] Registered new tserver with Master: bca70d4c5310482ba1379b5617714a67 (127.17.52.130:33013)
I20260927 03:19:25.686228 28839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:47051
W20260927 03:19:25.795725 29194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:19:25.796092 29194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:19:25.796190 29194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:19:25.797995 29194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:19:25.798123 29194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:19:25.798204 29194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:19:25.800691 29194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.52.148:35465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:19:25.801343 29194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:19:25.801671 29194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:19:25.808195 29200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.808555 29199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:19:25.808715 29202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:19:25.809615 29194 server_base.cc:1034] running on GCE node
I20260927 03:19:25.809835 29194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:19:25.810096 29194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:19:25.811256 29194 hybrid_clock.cc:648] HybridClock initialized: now 1790479165811227 us; error 43 us; skew 500 ppm
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:19:25.814589 29194 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:19:25.815094 29194 webserver.cc:460] Webserver started at http://127.17.52.131:42433/ using document root <none> and password file <none>
I20260927 03:19:25.815371 29194 fs_manager.cc:359] Metadata directory not provided
I20260927 03:19:25.815420 29194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:19:25.815543 29194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:25.836771 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "6254638fb4b04894a115bab19af683a6"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "a5219d57daae41d3003a5f70bd683519"
server_key_iv: "82050fa799c6abfdf4dd4bb034585107"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.837342 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "6254638fb4b04894a115bab19af683a6"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "a5219d57daae41d3003a5f70bd683519"
server_key_iv: "82050fa799c6abfdf4dd4bb034585107"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.842829 29194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260927 03:19:25.860776 29209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.861027 29194 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 03:19:25.861115 29194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "6254638fb4b04894a115bab19af683a6"
format_stamp: "Formatted at 2026-09-27 03:19:25 on dist-test-slave-7j43"
server_key: "a5219d57daae41d3003a5f70bd683519"
server_key_iv: "82050fa799c6abfdf4dd4bb034585107"
server_key_version: "kuduclusterkey@0"
I20260927 03:19:25.861217 29194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:19:25.891667 29194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:19:25.894129 29194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:19:25.894320 29194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:19:25.894569 29194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:19:25.894934 29194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:19:25.895012 29194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.895079 29194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:19:25.895133 29194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:19:25.901146 29194 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:38977
I20260927 03:19:25.901690 29194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:19:25.903491 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 29194
I20260927 03:19:25.903581 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:19:25.903770 17618 external_mini_cluster.cc:1428] Setting key 8f0bb77df0846bf92a10755a97421f33
I20260927 03:19:25.906045 29322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:38977 every 8 connection(s)
Sep 27 03:19:25 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:19:25.916860 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.906563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:43339 (local address 127.17.52.190:37041)
0927 03:19:25.906639 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.906641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.906834 (+ 193us) server_negotiation.cc:407] Connection header received
0927 03:19:25.907466 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.907468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.907479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.907508 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:19:25.908317 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.909142 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.910656 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.910925 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.910970 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.913583 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:19:25.913584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:19:25.913585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:19:25.913592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:19:25.915261 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.915743 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.915744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.915744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.915801 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.916100 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.916101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.916102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.916242 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:19:25.916285 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.916688 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.916769 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:19:25.917624 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.906293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:37041 (local address 127.17.52.131:43339)
0927 03:19:25.906712 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.906758 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.907351 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.907688 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.907693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.907921 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:19:25.908196 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.908202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.909268 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.909269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.910556 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.910559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.911066 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.911067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.911221 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.911816 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:19:25.911827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:19:25.913469 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:19:25.915351 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.915357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.915360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.915661 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.915876 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:19:25.915877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:19:25.915877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:19:25.916018 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:19:25.916332 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:19:25.916337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:19:25.916492 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.917219 (+ 727us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.917381 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":91,"threads_started":1}
I20260927 03:19:25.918257 29323 heartbeater.cc:347] Connected to a master server at 127.17.52.190:37041
I20260927 03:19:25.918350 29323 heartbeater.cc:464] Registering TS with master...
I20260927 03:19:25.918525 29323 heartbeater.cc:511] Master 127.17.52.190:37041 requested a full tablet report, sending...
I20260927 03:19:25.918915 28839 ts_manager.cc:194] Registered new tserver with Master: 6254638fb4b04894a115bab19af683a6 (127.17.52.131:38977)
I20260927 03:19:25.919411 28839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:43339
I20260927 03:19:25.926476 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:19:25.941174 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.932829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.17.52.190:37041)
0927 03:19:25.932873 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.932876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.933232 (+ 356us) server_negotiation.cc:407] Connection header received
0927 03:19:25.933334 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.933335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.933345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.933374 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:19:25.934170 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.934904 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.936007 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.936286 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.936331 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.937946 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:19:25.937947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:19:25.937948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:19:25.937954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:19:25.939620 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.940014 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.940016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.940016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.940069 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:19:25.940460 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:19:25.940461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:19:25.940461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:19:25.940691 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:19:25.940772 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.940992 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.941068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:19:25.943333 28839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:19:25.945668 28839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:19:25.957801 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38977 (local address 127.0.0.1:38888)
0927 03:19:25.950796 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.950804 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.950878 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.951180 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.951183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.951185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:19:25.951247 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.951249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.952891 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.952892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.956058 (+ 3166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.956060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.957513 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.957514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.957590 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.957604 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.957675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.957725 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":63,"threads_started":1}
I20260927 03:19:25.958240 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.17.52.131:38977)
0927 03:19:25.950575 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.950580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.950853 (+ 273us) server_negotiation.cc:407] Connection header received
0927 03:19:25.950968 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.950971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.950988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.951044 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:19:25.951334 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.952790 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.956183 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.957330 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.957378 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.957453 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.958102 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.958158 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":90,"threads_started":1}
I20260927 03:19:25.962095 29246 tablet_service.cc:1467] Processing CreateTablet for tablet 0270213ac9104d68a609ba75d98427a8 (DEFAULT_TABLE table=test-table [id=124608c277564d85b3a1f2b2afcfd03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:19:25.962495 29246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270213ac9104d68a609ba75d98427a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:19:25.968739 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46753 (local address 127.0.0.1:56538)
0927 03:19:25.950330 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.950369 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.950431 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.954664 (+ 4233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.954665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.954667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:19:25.954707 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.954709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.958554 (+ 3845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.958555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.959979 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.959982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.968499 (+ 8517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.968500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.968560 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.968564 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.968625 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.968664 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"thread_start_us":69,"threads_started":1}
I20260927 03:19:25.969321 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.17.52.129:46753)
0927 03:19:25.954498 (+ 5030us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.954504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.954514 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:19:25.954546 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.954548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.954564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.954773 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:19:25.954832 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.955870 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.960080 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.968405 (+ 8325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.969097 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.969181 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.969209 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.969252 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4995,"thread_start_us":40,"threads_started":1}
I20260927 03:19:25.970084 28978 tablet_service.cc:1467] Processing CreateTablet for tablet 0270213ac9104d68a609ba75d98427a8 (DEFAULT_TABLE table=test-table [id=124608c277564d85b3a1f2b2afcfd03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:19:25.970423 28978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270213ac9104d68a609ba75d98427a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:19:25.976543 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33013 (local address 127.0.0.1:51238)
0927 03:19:25.950623 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:25.950633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:25.950677 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:25.961301 (+ 10624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:25.961302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:25.961304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:19:25.961343 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.961344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.974167 (+ 12823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.974168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.975290 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:25.975293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.976331 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.976332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:25.976397 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.976402 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:19:25.976432 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:19:25.976470 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":87,"threads_started":1}
I20260927 03:19:25.977126 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:25.949391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.17.52.130:33013)
0927 03:19:25.960705 (+ 11314us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:25.960710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:25.960720 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:19:25.960757 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:25.960759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:25.960774 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:25.966966 (+ 6192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:19:25.967052 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.968102 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.975423 (+ 7321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:25.976232 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:25.976885 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:25.976970 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:25.976998 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 03:19:25.977036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11270,"thread_start_us":72,"threads_started":1}
I20260927 03:19:25.977633 29342 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Bootstrap starting.
I20260927 03:19:25.977880 29123 tablet_service.cc:1467] Processing CreateTablet for tablet 0270213ac9104d68a609ba75d98427a8 (DEFAULT_TABLE table=test-table [id=124608c277564d85b3a1f2b2afcfd03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:19:25.978217 29123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270213ac9104d68a609ba75d98427a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:19:25.978652 29343 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Bootstrap starting.
I20260927 03:19:25.979497 29342 tablet_bootstrap.cc:654] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Neither blocks nor log segments found. Creating new log.
I20260927 03:19:25.979815 29342 log.cc:824] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:19:25.980373 29343 tablet_bootstrap.cc:654] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Neither blocks nor log segments found. Creating new log.
I20260927 03:19:25.980688 29343 log.cc:824] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:19:25.981415 29342 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: No bootstrap required, opened a new log
I20260927 03:19:25.981475 29342 ts_tablet_manager.cc:1397] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 03:19:25.983985 29342 raft_consensus.cc:348] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.984263 29342 raft_consensus.cc:374] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:19:25.984356 29342 raft_consensus.cc:729] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6254638fb4b04894a115bab19af683a6, State: Initialized, Role: FOLLOWER
I20260927 03:19:25.984549 29342 consensus_queue.cc:260] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.984947 29342 ts_tablet_manager.cc:1428] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:19:25.985106 29323 heartbeater.cc:503] Master 127.17.52.190:37041 was elected leader, sending a full tablet report...
I20260927 03:19:25.986531 29343 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: No bootstrap required, opened a new log
I20260927 03:19:25.986654 29343 ts_tablet_manager.cc:1397] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260927 03:19:25.987371 29346 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Bootstrap starting.
I20260927 03:19:25.988421 29346 tablet_bootstrap.cc:654] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Neither blocks nor log segments found. Creating new log.
I20260927 03:19:25.988807 29346 log.cc:824] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Log is configured to *not* fsync() on all Append() calls
I20260927 03:19:25.989842 29343 raft_consensus.cc:348] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.990053 29343 raft_consensus.cc:374] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:19:25.990142 29343 raft_consensus.cc:729] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0300fdeb8edb4ad3ba4ffea758e7efa7, State: Initialized, Role: FOLLOWER
I20260927 03:19:25.990319 29343 consensus_queue.cc:260] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.990615 29343 ts_tablet_manager.cc:1428] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 03:19:25.990763 29051 heartbeater.cc:503] Master 127.17.52.190:37041 was elected leader, sending a full tablet report...
I20260927 03:19:25.993052 29346 tablet_bootstrap.cc:492] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: No bootstrap required, opened a new log
I20260927 03:19:25.993129 29346 ts_tablet_manager.cc:1397] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260927 03:19:25.995225 29346 raft_consensus.cc:348] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.995383 29346 raft_consensus.cc:374] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:19:25.995414 29346 raft_consensus.cc:729] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca70d4c5310482ba1379b5617714a67, State: Initialized, Role: FOLLOWER
I20260927 03:19:25.995523 29346 consensus_queue.cc:260] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:25.995744 29346 ts_tablet_manager.cc:1428] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:19:25.996124 29189 heartbeater.cc:503] Master 127.17.52.190:37041 was elected leader, sending a full tablet report...
Sep 27 03:19:26 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
W20260927 03:19:26.156041 29324 tablet.cc:2369] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:19:26.161226 29190 tablet.cc:2369] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:19:26.164045 29052 tablet.cc:2369] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:19:26 dist-test-slave-7j43 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479165, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:19:26.361501 29348 raft_consensus.cc:482] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:19:26.361644 29348 raft_consensus.cc:504] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:26.362022 29348 leader_election.cc:288] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6254638fb4b04894a115bab19af683a6 (127.17.52.131:38977), bca70d4c5310482ba1379b5617714a67 (127.17.52.130:33013)
I20260927 03:19:26.366935 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.362766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50751 (local address 127.17.52.131:38977)
0927 03:19:26.362860 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:26.362862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:26.363288 (+ 426us) server_negotiation.cc:407] Connection header received
0927 03:19:26.363392 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:26.363393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:26.363403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:26.363430 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:19:26.363670 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.364373 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.366022 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.366539 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.366590 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:26.366599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:26.366824 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 03:19:26.366864 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 03:19:26.367408 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.362269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38977 (local address 127.17.52.129:50751)
0927 03:19:26.363218 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:26.363231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:26.363318 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:26.363498 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:26.363499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:26.363515 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:19:26.363591 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:26.363593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.364488 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.364489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:26.365881 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:26.365884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.366671 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.366672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:26.366742 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:26.366746 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:19:26.367277 (+ 531us) client_negotiation.cc:240] Negotiation successful
0927 03:19:26.367322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":86,"threads_started":1}
I20260927 03:19:26.367885 29258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270213ac9104d68a609ba75d98427a8" candidate_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6254638fb4b04894a115bab19af683a6" is_pre_election: true
I20260927 03:19:26.368036 29258 raft_consensus.cc:2454] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0300fdeb8edb4ad3ba4ffea758e7efa7 in term 0.
I20260927 03:19:26.368343 28939 leader_election.cc:302] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0300fdeb8edb4ad3ba4ffea758e7efa7, 6254638fb4b04894a115bab19af683a6; no voters:
I20260927 03:19:26.368465 29348 raft_consensus.cc:2781] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:19:26.368501 29348 raft_consensus.cc:482] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:19:26.368527 29348 raft_consensus.cc:3037] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:19:26.369542 29348 raft_consensus.cc:504] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:26.369710 29348 leader_election.cc:288] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [CANDIDATE]: Term 1 election: Requested vote from peers 6254638fb4b04894a115bab19af683a6 (127.17.52.131:38977), bca70d4c5310482ba1379b5617714a67 (127.17.52.130:33013)
I20260927 03:19:26.370047 29258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270213ac9104d68a609ba75d98427a8" candidate_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6254638fb4b04894a115bab19af683a6"
I20260927 03:19:26.370116 29258 raft_consensus.cc:3037] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:19:26.370987 29258 raft_consensus.cc:2454] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0300fdeb8edb4ad3ba4ffea758e7efa7 in term 1.
I20260927 03:19:26.371196 28939 leader_election.cc:302] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0300fdeb8edb4ad3ba4ffea758e7efa7, 6254638fb4b04894a115bab19af683a6; no voters:
I20260927 03:19:26.371309 29348 raft_consensus.cc:2781] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:19:26.371439 29348 raft_consensus.cc:686] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [term 1 LEADER]: Becoming Leader. State: Replica: 0300fdeb8edb4ad3ba4ffea758e7efa7, State: Running, Role: LEADER
I20260927 03:19:26.371533 29348 consensus_queue.cc:237] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } }
I20260927 03:19:26.372311 28839 catalog_manager.cc:5495] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0300fdeb8edb4ad3ba4ffea758e7efa7 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 } health_report { overall_health: UNKNOWN } } }
I20260927 03:19:26.373610 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.362815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:34047 (local address 127.17.52.130:33013)
0927 03:19:26.362892 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:26.362894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:26.366370 (+ 3476us) server_negotiation.cc:407] Connection header received
0927 03:19:26.366927 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:26.366928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:26.366942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:26.366972 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:19:26.367202 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.367897 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.372366 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.372948 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.372999 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:26.373008 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:26.373485 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 03:19:26.373533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 03:19:26.373831 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.362464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33013 (local address 127.17.52.129:34047)
0927 03:19:26.366304 (+ 3840us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:26.366315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:26.366834 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:26.367041 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:19:26.367042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:19:26.367043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:19:26.367119 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:26.367121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.368428 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.368429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:26.369585 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:19:26.369588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.373070 (+ 3482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.373071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:19:26.373155 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:26.373159 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:19:26.373698 (+ 539us) client_negotiation.cc:240] Negotiation successful
0927 03:19:26.373740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3809,"thread_start_us":67,"threads_started":1}
I20260927 03:19:26.376266 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270213ac9104d68a609ba75d98427a8" candidate_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca70d4c5310482ba1379b5617714a67" is_pre_election: true
I20260927 03:19:26.376420 29143 raft_consensus.cc:2454] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0300fdeb8edb4ad3ba4ffea758e7efa7 in term 0.
I20260927 03:19:26.376545 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270213ac9104d68a609ba75d98427a8" candidate_uuid: "0300fdeb8edb4ad3ba4ffea758e7efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca70d4c5310482ba1379b5617714a67"
I20260927 03:19:26.376624 29142 raft_consensus.cc:3037] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:19:26.377236 29142 raft_consensus.cc:2454] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0300fdeb8edb4ad3ba4ffea758e7efa7 in term 1.
I20260927 03:19:26.397410 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.393859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.17.52.129:46753)
0927 03:19:26.393926 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:19:26.393929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:19:26.393936 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:19:26.393972 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:19:26.393973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:19:26.393995 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:19:26.394060 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:19:26.394245 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.395037 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.396252 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:19:26.396590 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:19:26.396667 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:19:26.396866 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:19:26.396998 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:19:26.397069 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:19:26.397276 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:19:26.397308 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:19:26.399518 29142 raft_consensus.cc:1261] T 0270213ac9104d68a609ba75d98427a8 P bca70d4c5310482ba1379b5617714a67 [term 1 FOLLOWER]: Refusing update from remote peer 0300fdeb8edb4ad3ba4ffea758e7efa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:19:26.399923 29348 consensus_queue.cc:1035] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bca70d4c5310482ba1379b5617714a67" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:19:26.402325 29258 raft_consensus.cc:1261] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 1 FOLLOWER]: Refusing update from remote peer 0300fdeb8edb4ad3ba4ffea758e7efa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:19:26.403093 29354 consensus_queue.cc:1035] T 0270213ac9104d68a609ba75d98427a8 P 0300fdeb8edb4ad3ba4ffea758e7efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6254638fb4b04894a115bab19af683a6" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:19:26.405817 29358 mvcc.cc:205] Tried to move back new op lower bound from 7333802665541509120 to 7333802665430351872. Current Snapshot: MvccSnapshot[applied={T|T < 7333802665541509120}]
I20260927 03:19:26.406052 29359 mvcc.cc:205] Tried to move back new op lower bound from 7333802665541509120 to 7333802665430351872. Current Snapshot: MvccSnapshot[applied={T|T < 7333802665541509120}]
I20260927 03:19:26.410589 29362 mvcc.cc:205] Tried to move back new op lower bound from 7333802665541509120 to 7333802665430351872. Current Snapshot: MvccSnapshot[applied={T|T < 7333802665541509120}]
I20260927 03:19:26.411803 28839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:19:26.411899 28839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:19:26.413862 28839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8b55d7e04cd4dae8d21b871f3e672c8: Sending DeleteTablet for 3 replicas of tablet 0270213ac9104d68a609ba75d98427a8
I20260927 03:19:26.414219 29246 tablet_service.cc:1514] Processing DeleteTablet for tablet 0270213ac9104d68a609ba75d98427a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:19:26 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:38888
I20260927 03:19:26.414808 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 28920
I20260927 03:19:26.416108 29123 tablet_service.cc:1514] Processing DeleteTablet for tablet 0270213ac9104d68a609ba75d98427a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:19:26 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:51238
I20260927 03:19:26.417132 29367 tablet_replica.cc:330] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: stopping tablet replica
I20260927 03:19:26.417250 29367 raft_consensus.cc:2229] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:19:26.417366 29367 raft_consensus.cc:2258] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:19:26.420852 28823 connection.cc:535] client connection to 127.17.52.129:46753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:19:26.420946 28823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:19:26.421722 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:19:26.421600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46753 (local address 127.0.0.1:56560)
0927 03:19:26.421644 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:26.421688 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:19:26.421921 28823 catalog_manager.cc:4584] TS 0300fdeb8edb4ad3ba4ffea758e7efa7 (127.17.52.129:46753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0270213ac9104d68a609ba75d98427a8: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46753: connect: Connection refused (error 111)
I20260927 03:19:26.422153 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 29059
I20260927 03:19:26.424515 29367 ts_tablet_manager.cc:1905] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:19:26.426183 29367 ts_tablet_manager.cc:1918] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:19:26.426316 29367 log.cc:1196] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/0270213ac9104d68a609ba75d98427a8
I20260927 03:19:26.426640 29367 ts_tablet_manager.cc:1939] T 0270213ac9104d68a609ba75d98427a8 P 6254638fb4b04894a115bab19af683a6: Deleting consensus metadata
I20260927 03:19:26.427121 28824 catalog_manager.cc:4841] TS 6254638fb4b04894a115bab19af683a6 (127.17.52.131:38977): tablet 0270213ac9104d68a609ba75d98427a8 (table test-table [id=124608c277564d85b3a1f2b2afcfd03e]) successfully deleted
W20260927 03:19:26.431456 29340 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:19:26.431144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:33013 (local address 127.0.0.1:51250)
0927 03:19:26.431190 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:19:26.431200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:19:26.431238 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:19:26.431417 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.130:33013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:19:26.431973 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 29194
I20260927 03:19:26.441486 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 28807
I20260927 03:19:26.447482 17618 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:19:26.781814 17618 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:19:26.781909 17618 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:19:27.126430 17618 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:19:27.126 UTC [28393] LOG: received smart shutdown request
2026-09-27 03:19:27.130 UTC [28393] LOG: background worker "logical replication launcher" (PID 28402) exited with exit code 1
2026-09-27 03:19:27.130 UTC [28397] LOG: shutting down
2026-09-27 03:19:27.280 UTC [28393] LOG: database system is shut down
2026-09-27T03:19:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46288 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:19:27 dist-test-slave-7j43 krb5kdc[29378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:19:27 dist-test-slave-7j43 krb5kdc[29378](info): set up 2 sockets
Sep 27 03:19:27 dist-test-slave-7j43 krb5kdc[29378](info): commencing operation
krb5kdc: starting...
W20260927 03:19:28.412979 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:19:28 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:19:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangeradmin_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangeradmin_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangerlookup_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangerlookup_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
I20260927 03:19:28.562832 17618 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRLekJh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/postgres -l logfile start
2026-09-27 03:19:30.790 UTC [29408] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:19:30.790 UTC [29408] LOG: listening on IPv4 address "127.17.52.148", port 53473
2026-09-27 03:19:30.793 UTC [29408] LOG: listening on Unix socket "/tmp/.s.PGSQL.53473"
2026-09-27 03:19:30.813 UTC [29411] LOG: database system was shut down at 2026-09-27 03:19:29 UTC
2026-09-27 03:19:30.816 UTC [29408] LOG: database system is ready to accept connections
I20260927 03:19:31.772905 17618 mini_postgres.cc:96] Postgres bound to 53473
2026-09-27 03:19:31.778 UTC [29419] FATAL: database "slave" does not exist
127.17.52.148:53473 - accepting connections
I20260927 03:19:31.779129 17618 mini_ranger.cc:161] Starting Ranger...
I20260927 03:19:31.793135 17618 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:19:32.208804 17618 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:19:32.208889 17618 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin
2026-09-27 03:19:32,479 [I] DB FLAVOR :POSTGRES
2026-09-27 03:19:32,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:19:32,479 [I] Checking connection..
2026-09-27 03:19:32,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:19:32,729 [I] Checking connection passed.
2026-09-27 03:19:32,729 [I] --------- Verifying version history table ---------
2026-09-27 03:19:32,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:19:32,991 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:19:32,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:19:33,230 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:19:33,231 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:19:33,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:19:33.461 UTC [29483] WARNING: there is no transaction in progress
2026-09-27 03:19:33,466 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:19:33,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:19:33,716 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:19:33,716 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:19:33,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:19:33,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:19:34,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7j43.c.gcp-upstream.internal','N') ;"
2026-09-27 03:19:34,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:19:34,711 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:19:34,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:19:34,953 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:19:34,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:19:35,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:19:35,439 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:19:35,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:19:36.330 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.347 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.363 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.377 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.395 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.414 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.476 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.482 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.495 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.505 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.515 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.528 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.536 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.542 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.548 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:36.554 UTC [29591] WARNING: there is no transaction in progress
2026-09-27 03:19:37,113 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:19:37,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:19:37,364 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:19:37,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:19:37,623 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:19:37,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:19:37,855 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:19:37,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:19:38,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:19:38,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7j43.c.gcp-upstream.internal';"
2026-09-27 03:19:38,536 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:19:38,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:19:38,768 [I] DB_PATCHES have already been applied
I20260927 03:19:38.773790 17618 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRLekJh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:19:38.773859 17618 mini_ranger.cc:193] Using host: 127.17.52.148
I20260927 03:19:38.776890 17618 mini_ranger.cc:227] Ranger admin URL: http://127.17.52.148:49005
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:19:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:19:39.867295 17618 mini_ranger.cc:160] Time spent starting Ranger: real 8.088s user 0.001s sys 0.007s
Sep 27, 2026 3:19:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:19:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:19:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:19:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:20:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:20:01.295331 17618 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.52.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.52.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangerkms_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.52.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/rangerkms_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.52.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.52.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.52.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/HTTP_127.17.52.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:20:01 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:20:01.393318 17618 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:20:01.403658 17618 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:20:02.200696 17618 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:20:02.200790 17618 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms
I20260927 03:20:02.200812 17618 mini_ranger_kms.cc:227] Using postgres at 127.17.52.148:53473
2026-09-27 03:20:02,362 [I] DB FLAVOR :POSTGRES
2026-09-27 03:20:02,362 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:20:02,362 [I] Checking connection
2026-09-27 03:20:02,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:20:02,589 [I] connection success
2026-09-27 03:20:02,590 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:20:02,590 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:20:02,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:20:02,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:20:02,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:20:02,822 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:20:02,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.52.148:53473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:20:03,068 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:20:03.534866 17618 mini_ranger_kms.cc:310] Created kms service
I20260927 03:20:03.749393 17618 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:20:03.794443 17618 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:20:03 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:04.173285 17618 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:20:04.173414 17618 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRLekJh/build/release/bin/postgresql.jar:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRLekJh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:20:04.173435 17618 mini_ranger_kms.cc:240] Using host: 127.17.52.148
I20260927 03:20:04.178134 17618 mini_ranger_kms.cc:276] Ranger KMS PID: 29771
I20260927 03:20:04.178262 17618 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.52.148:42873
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:20:06.449083 17618 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.056s user 0.002s sys 0.143s
I20260927 03:20:06.449191 17618 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:20:06.449236 17618 mini_ranger_kms.cc:401] 127.17.52.148:42873/kms/v1/keys
Sep 27 03:20:06 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479206, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:20:07 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479206, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
Sep 27 03:20:07 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479206, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.52.148@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
Sep 27 03:20:09 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:20:09.479050 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38033
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38033
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:09.575445 29822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:09.575647 29822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:09.575668 29822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:09.576871 29822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:09.576911 29822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:09.576925 29822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:09.576984 29822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:09.577003 29822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:09.578680 29822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.52.148:42873/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38033
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:09.578928 29822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:09.579150 29822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:09.584110 29830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:09.584138 29822 server_base.cc:1034] running on GCE node
W20260927 03:20:09.584149 29827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:09.584110 29828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:09.584627 29822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:09.584935 29822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:09.586113 29822 hybrid_clock.cc:648] HybridClock initialized: now 1790479209586096 us; error 30 us; skew 500 ppm
Sep 27 03:20:09 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:09.593652 29822 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:09.594215 29822 webserver.cc:460] Webserver started at http://127.17.52.190:42603/ using document root <none> and password file <none>
I20260927 03:20:09.594481 29822 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:09.594539 29822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:09.594681 29822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:20:09 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:09.647560 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "833206ac25654d4891d618244f3859a9"
format_stamp: "Formatted at 2026-09-27 03:20:09 on dist-test-slave-7j43"
server_key: "d67addd48f9087451e33232ad54bbf44"
server_key_iv: "e4ae1a7ae587c22d7fa1a672e8f7df96"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:09.648054 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "833206ac25654d4891d618244f3859a9"
format_stamp: "Formatted at 2026-09-27 03:20:09 on dist-test-slave-7j43"
server_key: "d67addd48f9087451e33232ad54bbf44"
server_key_iv: "e4ae1a7ae587c22d7fa1a672e8f7df96"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:09.649909 29822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 03:20:09.663770 29837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:09.664024 29822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 03:20:09.664140 29822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "833206ac25654d4891d618244f3859a9"
format_stamp: "Formatted at 2026-09-27 03:20:09 on dist-test-slave-7j43"
server_key: "d67addd48f9087451e33232ad54bbf44"
server_key_iv: "e4ae1a7ae587c22d7fa1a672e8f7df96"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:09.664242 29822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:09.686321 29822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:09.688346 29822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:09.688490 29822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:09.692417 29822 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38033
I20260927 03:20:09.692499 29889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38033 every 8 connection(s)
I20260927 03:20:09.692868 29822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:09.693660 29890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:09.695178 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 29822
I20260927 03:20:09.695286 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:20:09.695451 17618 external_mini_cluster.cc:1428] Setting key fc50f7fea5baad6f34190900ff61956e
I20260927 03:20:09.696668 29890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Bootstrap starting.
I20260927 03:20:09.697466 29890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:09.697791 29890 log.cc:824] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:09.698568 29890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: No bootstrap required, opened a new log
I20260927 03:20:09.701010 29890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:09.701192 29890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:09.701226 29890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833206ac25654d4891d618244f3859a9, State: Initialized, Role: FOLLOWER
I20260927 03:20:09.701373 29890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:09.701445 29890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:09.701479 29890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:09.701519 29890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:20:09 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:09.702170 29890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:09.702273 29890 leader_election.cc:302] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833206ac25654d4891d618244f3859a9; no voters:
I20260927 03:20:09.702432 29890 leader_election.cc:288] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:20:09.702548 29895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:09.702687 29890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:09.702803 29895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 LEADER]: Becoming Leader. State: Replica: 833206ac25654d4891d618244f3859a9, State: Running, Role: LEADER
I20260927 03:20:09.702955 29895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:09.703363 29895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833206ac25654d4891d618244f3859a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } } }
I20260927 03:20:09.703503 29895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:09.703698 29895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833206ac25654d4891d618244f3859a9. Latest consensus state: current_term: 1 leader_uuid: "833206ac25654d4891d618244f3859a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } } }
I20260927 03:20:09.703819 29895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:09.704035 29890 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:20:09.707114 29906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:09.708385 29906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:09.709067 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:09.695942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.17.52.190:38033)
0927 03:20:09.696130 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:09.696139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:09.696172 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:20:09.696558 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:09.696609 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:09.696668 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:09.696832 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:09.697470 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:09.698637 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:09.699450 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:09.699852 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:09.699966 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:09.702438 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:09.702443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:09.702444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:09.702458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:09.704433 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:09.705921 (+ 1488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:09.705986 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:09.705987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:09.706494 (+ 507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:09.706758 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:09.706759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:09.706760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:09.707580 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:09.708298 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:09.708390 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:20:09.708745 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260927 03:20:09.713073 29906 catalog_manager.cc:1347] Generated new cluster ID: caa9a2bca8294fbbac2932ae2d22eca3
I20260927 03:20:09.713150 29906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:20:09.721937 29906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:20:09.722657 29906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:09.729681 29906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Generated new TSK 0
I20260927 03:20:09.729849 29906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:20:10.398857 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:10.515766 29935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:10.516038 29935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:10.516072 29935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:10.517764 29935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:10.517822 29935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:10.517860 29935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:10.519961 29935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:10.520208 29935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:10.520495 29935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:10.527580 29940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:10.527638 29941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:10.527978 29943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:10.528543 29935 server_base.cc:1034] running on GCE node
I20260927 03:20:10.528782 29935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:10.529045 29935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:10.530221 29935 hybrid_clock.cc:648] HybridClock initialized: now 1790479210530199 us; error 33 us; skew 500 ppm
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:10.533440 29935 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:10.533927 29935 webserver.cc:460] Webserver started at http://127.17.52.129:35495/ using document root <none> and password file <none>
I20260927 03:20:10.534168 29935 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:10.534222 29935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:10.534359 29935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:10.549603 29935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "2331ea232bae49a0ac2925ca21752861"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c69b76f4e40f48ee0dc1f537402202c4"
server_key_iv: "01fe3d11ac1f0f3d73525d38393e52c5"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.550153 29935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "2331ea232bae49a0ac2925ca21752861"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c69b76f4e40f48ee0dc1f537402202c4"
server_key_iv: "01fe3d11ac1f0f3d73525d38393e52c5"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.560997 29935 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260927 03:20:10.574752 29952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.574926 29935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:10.575024 29935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "2331ea232bae49a0ac2925ca21752861"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c69b76f4e40f48ee0dc1f537402202c4"
server_key_iv: "01fe3d11ac1f0f3d73525d38393e52c5"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.575119 29935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:10.599169 29935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:10.601547 29935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:10.601704 29935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:10.601967 29935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:10.602342 29935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:10.602380 29935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.602420 29935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:10.602447 29935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.609800 29935 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:33565
I20260927 03:20:10.609897 30065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:33565 every 8 connection(s)
I20260927 03:20:10.610280 29935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:10.616045 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 29935
I20260927 03:20:10.616164 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:10.616381 17618 external_mini_cluster.cc:1428] Setting key ecb15cdece2562c427ebdf1d6a0828ee
I20260927 03:20:10.622678 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:10.611013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.129:46447)
0927 03:20:10.611183 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:10.611227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:10.611726 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:10.612006 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:10.612010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:10.612197 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:10.612466 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:10.612471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.613201 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.613203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:10.614027 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:10.614029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.614466 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.614468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:10.614583 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:10.615131 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:10.615139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:10.616657 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:10.620883 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:10.620891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:10.620893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:10.621224 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:10.621467 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:10.621468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:10.621468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:10.621615 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:10.621930 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:10.621936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:10.622111 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:20:10.622208 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:20:10.622383 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260927 03:20:10.623231 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:10.611042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:46447 (local address 127.17.52.190:38033)
0927 03:20:10.611290 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:10.611292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:10.611303 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:20:10.611847 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:10.611848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:10.611863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:10.611933 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:10.612560 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.613104 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.614129 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.614362 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.614434 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:10.616749 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:10.616750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:10.616751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:10.616758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:10.620024 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:10.621322 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:10.621323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:10.621324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:10.621391 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:10.621694 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:10.621695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:10.621695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:10.621846 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:10.622955 (+ 1109us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:10.622997 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:20:10.623089 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260927 03:20:10.623595 30066 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:10.623685 30066 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:10.623908 30066 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:10.624464 29854 ts_manager.cc:194] Registered new tserver with Master: 2331ea232bae49a0ac2925ca21752861 (127.17.52.129:33565)
I20260927 03:20:10.625423 29854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:46447
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:20:10.644948 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:20:10.761175 30072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:10.761456 30072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:10.761492 30072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:10.763159 30072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:10.763217 30072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:10.763255 30072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:10.765388 30072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:10.765614 30072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:10.765851 30072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:10.772629 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:10.772917 30072 server_base.cc:1034] running on GCE node
W20260927 03:20:10.773030 30082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:10.772645 30080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:10.773342 30072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:10.773608 30072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:10.774781 30072 hybrid_clock.cc:648] HybridClock initialized: now 1790479210774753 us; error 39 us; skew 500 ppm
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:10.777848 30072 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:10.778319 30072 webserver.cc:460] Webserver started at http://127.17.52.130:45299/ using document root <none> and password file <none>
I20260927 03:20:10.778578 30072 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:10.778632 30072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:10.778769 30072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:10.791420 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "ba68c340714c4aafb25b9da0d512b639"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c2910b9e3ae79a47dad458b97387b1e0"
server_key_iv: "2f906f58f7c2e83c4f8aee02a0e39ba0"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.791829 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "ba68c340714c4aafb25b9da0d512b639"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c2910b9e3ae79a47dad458b97387b1e0"
server_key_iv: "2f906f58f7c2e83c4f8aee02a0e39ba0"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.793411 30072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:20:10.808691 30090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.809016 30072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:10.809181 30072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "ba68c340714c4aafb25b9da0d512b639"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c2910b9e3ae79a47dad458b97387b1e0"
server_key_iv: "2f906f58f7c2e83c4f8aee02a0e39ba0"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:10.809361 30072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:10.834882 30072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:10.837292 30072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:10.837440 30072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:10.837741 30072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:10.838119 30072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:10.838174 30072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.838209 30072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:10.838233 30072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:10.845641 30072 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:46381
I20260927 03:20:10.845749 30203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:46381 every 8 connection(s)
I20260927 03:20:10.846166 30072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:20:10.852514 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30072
I20260927 03:20:10.852631 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:20:10.852833 17618 external_mini_cluster.cc:1428] Setting key e8bb21b410cdb06df0fe729359ad9bca
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:10.859937 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:10.846917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.130:33195)
0927 03:20:10.847085 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:10.847135 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:10.847818 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:10.848126 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:10.848130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:10.848371 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:10.848695 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:10.848703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.849750 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.849752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:10.851037 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:10.851040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.851636 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.851637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:10.851788 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:10.852405 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:10.852416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:10.856187 (+ 3771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:10.858159 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:10.858167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:10.858169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:10.858488 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:10.858718 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:10.858719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:10.858719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:10.858863 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:10.859192 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:10.859199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:10.859376 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:20:10.859506 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 03:20:10.859671 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260927 03:20:10.860390 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:10.846939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:33195 (local address 127.17.52.190:38033)
0927 03:20:10.846989 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:10.846991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:10.847225 (+ 234us) server_negotiation.cc:407] Connection header received
0927 03:20:10.847947 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:10.847949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:10.847967 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:10.848032 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:10.848805 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.849631 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.851143 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:10.851475 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:10.851525 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:10.856313 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:10.856314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:10.856315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:10.856322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:10.858063 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:10.858577 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:10.858578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:10.858578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:10.858637 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:10.858958 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:10.858958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:10.858959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:10.859096 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:10.859164 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:10.860200 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 03:20:10.860301 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:20:10.860841 30204 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:10.860940 30204 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:10.861138 30204 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:10.861558 29854 ts_manager.cc:194] Registered new tserver with Master: ba68c340714c4aafb25b9da0d512b639 (127.17.52.130:46381)
I20260927 03:20:10.862046 29854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:33195
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:20:10.887435 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:10.979750 30209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:10.979982 30209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:10.980023 30209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:10.981547 30209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:10.981603 30209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:10.981642 30209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:10.983588 30209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:10.983861 30209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:10.984086 30209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:10.989617 30214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:10.989849 30217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:10.989858 30215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:10.990579 30209 server_base.cc:1034] running on GCE node
I20260927 03:20:10.990794 30209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:10.991046 30209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:10.992225 30209 hybrid_clock.cc:648] HybridClock initialized: now 1790479210992200 us; error 40 us; skew 500 ppm
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:10.995107 30209 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:10.995527 30209 webserver.cc:460] Webserver started at http://127.17.52.131:40629/ using document root <none> and password file <none>
I20260927 03:20:10.995776 30209 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:10.995847 30209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:10.995995 30209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:20:10 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:11.009994 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "aedad9c8a86c41dcac1d761541fef2d3"
format_stamp: "Formatted at 2026-09-27 03:20:11 on dist-test-slave-7j43"
server_key: "2d8b7a625b8b1144df8b616615ddd555"
server_key_iv: "94edc227d14a00039f38f003b8792cd6"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.010592 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "aedad9c8a86c41dcac1d761541fef2d3"
format_stamp: "Formatted at 2026-09-27 03:20:11 on dist-test-slave-7j43"
server_key: "2d8b7a625b8b1144df8b616615ddd555"
server_key_iv: "94edc227d14a00039f38f003b8792cd6"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.017134 30209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.001s
I20260927 03:20:11.030078 30224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.030256 30209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:11.030352 30209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "aedad9c8a86c41dcac1d761541fef2d3"
format_stamp: "Formatted at 2026-09-27 03:20:11 on dist-test-slave-7j43"
server_key: "2d8b7a625b8b1144df8b616615ddd555"
server_key_iv: "94edc227d14a00039f38f003b8792cd6"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.030450 30209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:11.058843 30209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:11.061239 30209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:11.061384 30209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:11.061615 30209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:11.061970 30209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:11.062011 30209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.062067 30209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:11.062085 30209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.068087 30209 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:34351
I20260927 03:20:11.068502 30209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:20:11.069362 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30209
I20260927 03:20:11.069482 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:20:11.069710 17618 external_mini_cluster.cc:1428] Setting key 07a1504871a13b6ef5a14b4c3ff7ff7f
I20260927 03:20:11.070994 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:34351 every 8 connection(s)
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479210, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:11.081353 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.071766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:34113 (local address 127.17.52.190:38033)
0927 03:20:11.071809 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:11.071811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:11.072113 (+ 302us) server_negotiation.cc:407] Connection header received
0927 03:20:11.072663 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:11.072664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:11.072677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:11.072723 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:11.073276 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.073908 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.075293 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.075594 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.075636 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.077994 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:11.077996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:11.077996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:11.078003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:11.079683 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.080188 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.080190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.080190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.080256 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.080512 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.080513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.080514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.080672 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:11.080774 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:11.081126 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 03:20:11.081224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:20:11.081893 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.071784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.131:34113)
0927 03:20:11.071958 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:11.072000 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:11.072523 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:11.072802 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:11.072805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:11.072986 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:11.073178 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.073182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.074022 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.074023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.075190 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.075193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.075766 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.075767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.075903 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.076509 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:11.076519 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:11.077891 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:11.079817 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.079825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.079827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.080094 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.080324 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.080325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.080325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.080443 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.080767 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:11.080773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:11.080973 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 03:20:11.081482 (+ 509us) client_negotiation.cc:240] Negotiation successful
0927 03:20:11.081655 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260927 03:20:11.082432 30338 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:11.082530 30338 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.082715 30338 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:11.083109 29854 ts_manager.cc:194] Registered new tserver with Master: aedad9c8a86c41dcac1d761541fef2d3 (127.17.52.131:34351)
I20260927 03:20:11.083526 29854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:34113
I20260927 03:20:11.092623 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:11.093235 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 29935
I20260927 03:20:11.099486 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30072
I20260927 03:20:11.104641 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30209
I20260927 03:20:11.110729 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 29822
I20260927 03:20:11.115338 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38033
--webserver_interface=127.17.52.190
--webserver_port=42603
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38033
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:11.195849 30341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:11.196055 30341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:11.196076 30341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:11.197264 30341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:11.197309 30341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:11.197324 30341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:11.197376 30341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:11.197402 30341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:11.199275 30341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38033
--ranger_config_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:11.199517 30341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:11.199756 30341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:20:11.204612 30341 server_base.cc:1034] running on GCE node
W20260927 03:20:11.204532 30347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.204568 30346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.204532 30349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:11.205010 30341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:11.205310 30341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:11.206467 30341 hybrid_clock.cc:648] HybridClock initialized: now 1790479211206452 us; error 42 us; skew 500 ppm
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:11.208880 30341 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:11.209244 30341 webserver.cc:460] Webserver started at http://127.17.52.190:42603/ using document root <none> and password file <none>
I20260927 03:20:11.209430 30341 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:11.209477 30341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:11.244771 30341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:11.245466 30357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.245672 30341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:11.245764 30341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "833206ac25654d4891d618244f3859a9"
format_stamp: "Formatted at 2026-09-27 03:20:09 on dist-test-slave-7j43"
server_key: "d67addd48f9087451e33232ad54bbf44"
server_key_iv: "e4ae1a7ae587c22d7fa1a672e8f7df96"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.246104 30341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:11.252292 30341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:11.253803 30341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:11.253906 30341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:11.257836 30341 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38033
I20260927 03:20:11.257912 30409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38033 every 8 connection(s)
I20260927 03:20:11.258260 30341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:11.259377 30410 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:20:11.259995 30410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Bootstrap starting.
I20260927 03:20:11.260640 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30341
I20260927 03:20:11.260962 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:33565
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:38033
--webserver_port=35495
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:11.268368 30410 log.cc:824] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:11.270007 30410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:11.270292 30410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Bootstrap complete.
I20260927 03:20:11.272707 30410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:11.272941 30410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833206ac25654d4891d618244f3859a9, State: Initialized, Role: FOLLOWER
I20260927 03:20:11.273100 30410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:11.273165 30410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:11.273211 30410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:11.273257 30410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:20:11.274199 30410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:11.274310 30410 leader_election.cc:302] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833206ac25654d4891d618244f3859a9; no voters:
I20260927 03:20:11.274477 30410 leader_election.cc:288] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:20:11.274580 30414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:20:11.274691 30414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [term 2 LEADER]: Becoming Leader. State: Replica: 833206ac25654d4891d618244f3859a9, State: Running, Role: LEADER
I20260927 03:20:11.274775 30410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:11.274783 30414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } }
I20260927 03:20:11.275097 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833206ac25654d4891d618244f3859a9. Latest consensus state: current_term: 2 leader_uuid: "833206ac25654d4891d618244f3859a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } } }
I20260927 03:20:11.275184 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:11.275568 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "833206ac25654d4891d618244f3859a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833206ac25654d4891d618244f3859a9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38033 } } }
I20260927 03:20:11.275652 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:11.275703 30418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:11.276165 30418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:11.276284 30418 catalog_manager.cc:1259] Loaded cluster ID: caa9a2bca8294fbbac2932ae2d22eca3
I20260927 03:20:11.276319 30418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:20:11.277076 30410 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:20:11.277570 30418 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:11.280140 30418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Loaded TSK: 0
I20260927 03:20:11.280740 30418 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:20:11.348855 30412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:11.349150 30412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:11.349187 30412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:11.350716 30412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:11.350808 30412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:11.350857 30412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:11.353022 30412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=35495
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:11.353274 30412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:11.353541 30412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:11.359642 30447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.359663 30448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.359927 30450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:11.360975 30412 server_base.cc:1034] running on GCE node
I20260927 03:20:11.361174 30412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:11.361420 30412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:11.364642 30412 hybrid_clock.cc:648] HybridClock initialized: now 1790479211364623 us; error 27 us; skew 500 ppm
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:11.367830 30412 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:11.368337 30412 webserver.cc:460] Webserver started at http://127.17.52.129:35495/ using document root <none> and password file <none>
I20260927 03:20:11.368611 30412 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:11.368698 30412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:11.396375 30412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 03:20:11.397750 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.398600 30412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260927 03:20:11.398708 30412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "2331ea232bae49a0ac2925ca21752861"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c69b76f4e40f48ee0dc1f537402202c4"
server_key_iv: "01fe3d11ac1f0f3d73525d38393e52c5"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.399174 30412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:11.427856 30412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:11.430418 30412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:11.430583 30412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:11.430872 30412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:11.431262 30412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:11.431322 30412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.431360 30412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:11.431382 30412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.438937 30412 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:33565
I20260927 03:20:11.439086 30571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:33565 every 8 connection(s)
I20260927 03:20:11.439415 30412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:11.443466 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30412
I20260927 03:20:11.443815 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:46381
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:38033
--webserver_port=45299
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:11.454588 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.440884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.129:56503)
0927 03:20:11.441141 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:11.441183 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:11.441788 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:11.442546 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:11.442550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:11.442775 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:11.443030 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.443036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.444742 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.444743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.445981 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.445984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.446909 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.446910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.447026 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.447630 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:11.447640 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:11.449377 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:11.452675 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.452682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.452684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.452995 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.453296 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.453297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.453297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.453432 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.453917 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:11.453922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:11.454081 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 03:20:11.454175 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:20:11.454348 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":83,"threads_started":1}
I20260927 03:20:11.455211 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.440975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:56503 (local address 127.17.52.190:38033)
0927 03:20:11.442001 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:11.442009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:11.442047 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:20:11.442198 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:11.442205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:11.442261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:11.442390 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:11.443162 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.444620 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.446091 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.446708 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.446802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.449511 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:11.449517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:11.449517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:11.449528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:11.452550 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.453100 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.453103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.453104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.453209 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.453514 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.453514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.453515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.453825 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:11.454766 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:11.454839 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:20:11.454964 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":82,"threads_started":1}
I20260927 03:20:11.455940 30572 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:11.456029 30572 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.456208 30572 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.457146 30572 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:38033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:20:11.457233 30572 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.457309 30572 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:11.457710 30572 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.457778 30572 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.458089 30572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:20:11.569229 30576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:11.569551 30576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:11.569592 30576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:11.571382 30576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:11.571444 30576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:11.571486 30576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:11.573711 30576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:46381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=45299
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:11.573973 30576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:11.574258 30576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:11.579895 30581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.579938 30584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:11.580255 30576 server_base.cc:1034] running on GCE node
W20260927 03:20:11.580466 30582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:11.580729 30576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:11.581051 30576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:11.582238 30576 hybrid_clock.cc:648] HybridClock initialized: now 1790479211582225 us; error 28 us; skew 500 ppm
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:11.585276 30576 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:11.585893 30576 webserver.cc:460] Webserver started at http://127.17.52.130:45299/ using document root <none> and password file <none>
I20260927 03:20:11.586205 30576 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:11.586308 30576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:11.605439 30576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:11.606406 30592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.606637 30576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:11.606760 30576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "ba68c340714c4aafb25b9da0d512b639"
format_stamp: "Formatted at 2026-09-27 03:20:10 on dist-test-slave-7j43"
server_key: "c2910b9e3ae79a47dad458b97387b1e0"
server_key_iv: "2f906f58f7c2e83c4f8aee02a0e39ba0"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.607241 30576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:11.632870 30576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:11.635135 30576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:11.635305 30576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:11.635586 30576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:11.636006 30576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:11.636065 30576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.636098 30576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:11.636122 30576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.643632 30576 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:46381
I20260927 03:20:11.643784 30705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:46381 every 8 connection(s)
I20260927 03:20:11.644121 30576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:11.654100 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30576
I20260927 03:20:11.654328 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.52.148:42873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:34351
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:38033
--webserver_port=40629
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:11.656543 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.645497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:41843 (local address 127.17.52.190:38033)
0927 03:20:11.645550 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:11.645552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:11.645852 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:20:11.646495 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:11.646497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:11.646514 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:11.646589 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:11.647358 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.647876 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.649454 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.649799 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.649849 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.652454 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:11.652456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:11.652456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:11.652464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:11.654708 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.655445 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.655446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.655447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.655506 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.655810 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.655811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.655812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.655934 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:11.656016 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:11.656361 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 03:20:11.656447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:20:11.657094 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.645461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.130:41843)
0927 03:20:11.645707 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:11.645748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:11.646362 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:11.646724 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:11.646728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:11.646970 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:11.647266 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.647275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.647978 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.647980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.649329 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.649332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.649959 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.649960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.650100 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.650691 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:11.650701 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:11.652343 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:11.655025 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.655032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.655034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.655346 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.655587 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.655587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.655588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.655726 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.656013 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:11.656019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:11.656179 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 03:20:11.656696 (+ 517us) client_negotiation.cc:240] Negotiation successful
0927 03:20:11.656860 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260927 03:20:11.657671 30706 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:11.657797 30706 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.657981 30706 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.658639 30706 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:38033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:20:11.658735 30706 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.658806 30706 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:11.659090 30706 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.659166 30706 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.659440 30706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:20:11.769731 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:11.770022 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:11.770076 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:11.771768 30709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:11.771835 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:11.771883 30709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:11.774026 30709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.52.148:42873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=40629
--tserver_master_addrs=127.17.52.190:38033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:11.774322 30709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:11.774593 30709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:11.781251 30716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.781316 30718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:11.781522 30715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:11.781994 30709 server_base.cc:1034] running on GCE node
I20260927 03:20:11.782207 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:11.782486 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:11.783598 30709 hybrid_clock.cc:648] HybridClock initialized: now 1790479211783545 us; error 64 us; skew 500 ppm
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:11.787187 30709 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:11.787700 30709 webserver.cc:460] Webserver started at http://127.17.52.131:40629/ using document root <none> and password file <none>
I20260927 03:20:11.787957 30709 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:11.788023 30709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:11.812258 30709 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260927 03:20:11.813658 30726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.813925 30709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:11.814040 30709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "aedad9c8a86c41dcac1d761541fef2d3"
format_stamp: "Formatted at 2026-09-27 03:20:11 on dist-test-slave-7j43"
server_key: "2d8b7a625b8b1144df8b616615ddd555"
server_key_iv: "94edc227d14a00039f38f003b8792cd6"
server_key_version: "kuduclusterkey@0"
I20260927 03:20:11.814553 30709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:11.830513 30709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:11.832839 30709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:11.833001 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:11.833259 30709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:11.833624 30709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:11.833669 30709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.833703 30709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:11.833720 30709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:11.840899 30709 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:34351
I20260927 03:20:11.840978 30839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:34351 every 8 connection(s)
I20260927 03:20:11.841439 30709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:20:11 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479211, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:11.851495 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30709
I20260927 03:20:11.852982 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.842251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:46147 (local address 127.17.52.190:38033)
0927 03:20:11.842328 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:11.842330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:11.842544 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:20:11.843065 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:11.843066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:11.843080 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:11.843164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:11.843811 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.844622 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.845919 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.846207 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.846291 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.849660 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:11.849662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:11.849662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:11.849671 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:11.851285 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.851817 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.851818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.851818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.851889 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.852213 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.852214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.852214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.852375 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:11.852417 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:11.852809 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 03:20:11.852892 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:20:11.853199 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.842243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:38033 (local address 127.17.52.131:46147)
0927 03:20:11.842394 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:11.842439 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:11.842957 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:11.843223 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:11.843227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:11.843412 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:11.843721 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.843728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.844705 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.844706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.845819 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:11.845822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.846336 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.846337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:11.846485 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.847136 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:11.847146 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:11.849567 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:11.851398 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.851405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.851407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.851718 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.851982 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:11.851983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:11.851983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:11.852128 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:11.852461 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:11.852467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:11.852655 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 03:20:11.852783 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:20:11.852939 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 03:20:11.853629 30840 heartbeater.cc:347] Connected to a master server at 127.17.52.190:38033
I20260927 03:20:11.853742 30840 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.853947 30840 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.854651 30840 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:38033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:20:11.854724 30840 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.854800 30840 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:11.855163 30840 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:11.855237 30840 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
W20260927 03:20:11.855568 30840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:20:11.865015 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:11.851948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.17.52.190:38033)
0927 03:20:11.852280 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:11.852282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:11.852288 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:20:11.852305 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:11.852306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:11.852317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:11.852383 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:11.854486 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.855411 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.856707 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:11.857017 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:11.857094 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:11.858565 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:11.858566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:11.858567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:11.858573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:11.861273 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.864049 (+ 2776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.864050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.864051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.864121 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:11.864367 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:11.864368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:11.864369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:11.864512 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:11.864742 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:11.864785 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 03:20:11.864898 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":65,"threads_started":1}
Sep 27 03:20:12 dist-test-slave-7j43 krb5kdc[29378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479212, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:12.458273 30572 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:12.458441 30572 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:12.458861 30374 ts_manager.cc:194] Registered new tserver with Master: 2331ea232bae49a0ac2925ca21752861 (127.17.52.129:33565)
I20260927 03:20:12.459911 30374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:56503
I20260927 03:20:12.659616 30706 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:12.659775 30706 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:12.660046 30374 ts_manager.cc:194] Registered new tserver with Master: ba68c340714c4aafb25b9da0d512b639 (127.17.52.130:46381)
I20260927 03:20:12.660627 30374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:41843
Sep 27 03:20:12 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479212, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:12.855758 30840 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:12.855916 30840 heartbeater.cc:511] Master 127.17.52.190:38033 requested a full tablet report, sending...
I20260927 03:20:12.856251 30374 ts_manager.cc:194] Registered new tserver with Master: aedad9c8a86c41dcac1d761541fef2d3 (127.17.52.131:34351)
I20260927 03:20:12.856798 30374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:46147
I20260927 03:20:12.863891 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:12.874948 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.864891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.17.52.190:38033)
0927 03:20:12.865007 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.865009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.867546 (+ 2537us) server_negotiation.cc:407] Connection header received
0927 03:20:12.867620 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.867621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.867635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.867706 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:12.868390 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.869198 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.870165 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.870469 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.870542 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.871906 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:12.871907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:12.871908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:12.871916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:12.873699 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:12.874071 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:12.874073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:12.874073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:12.874191 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:12.874400 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:12.874401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:12.874401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:12.874533 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:12.874638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.874765 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.874850 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260927 03:20:12.877163 30374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:20:12.879338 30374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:20:12.887913 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:46381 (local address 127.0.0.1:52232)
0927 03:20:12.883547 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.883569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:12.883610 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:12.883935 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:12.883935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:12.883937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:12.883975 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.883976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.885450 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.885451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.886575 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.886578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.887612 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.887613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.887694 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.887708 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:20:12.887779 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:20:12.887832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":87,"threads_started":1}
I20260927 03:20:12.888279 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.17.52.129:33565)
0927 03:20:12.882704 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.882709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.883286 (+ 577us) server_negotiation.cc:407] Connection header received
0927 03:20:12.883338 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.883341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.883357 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.883440 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:12.883632 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.884709 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.886534 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.887671 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.887742 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.887796 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.888143 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.888206 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260927 03:20:12.888885 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.17.52.130:46381)
0927 03:20:12.883722 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.883726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.883735 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:20:12.883767 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.883769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.883783 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.883856 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:12.884052 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.885341 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.886671 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.887430 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.887499 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.887558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.888762 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.888807 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":76,"threads_started":1}
I20260927 03:20:12.888069 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33565 (local address 127.0.0.1:52268)
0927 03:20:12.883084 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.883121 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:12.883195 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:12.883456 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:12.883459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:12.883462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:12.883533 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.883534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.884825 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.884826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.886433 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.886435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.887831 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.887832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.887902 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.887906 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:12.887956 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:20:12.887995 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":78,"threads_started":1}
I20260927 03:20:12.889889 30495 tablet_service.cc:1467] Processing CreateTablet for tablet 11a3f160a64244348a57733bad8516de (DEFAULT_TABLE table=test-table [id=007ea502c0634d569a9de08055436da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:12.890282 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a3f160a64244348a57733bad8516de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:12.892813 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34351 (local address 127.0.0.1:47596)
0927 03:20:12.888285 (+ 5750us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.888294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:12.888344 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:12.888634 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:12.888635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:12.888636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:12.888662 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.888664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.890224 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.890225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.891502 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.891506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.892560 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.892561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.892655 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.892658 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:12.892706 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:20:12.892746 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5740,"thread_start_us":55,"threads_started":1}
I20260927 03:20:12.893067 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.882914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.17.52.131:34351)
0927 03:20:12.883291 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.883296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.888406 (+ 5110us) server_negotiation.cc:407] Connection header received
0927 03:20:12.888445 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.888448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.888463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.888532 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:12.888752 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.889845 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.891620 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.892407 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.892476 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.892528 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.892949 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.892993 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":62,"threads_started":1}
I20260927 03:20:12.893744 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 11a3f160a64244348a57733bad8516de (DEFAULT_TABLE table=test-table [id=007ea502c0634d569a9de08055436da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:12.893949 30873 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Bootstrap starting.
I20260927 03:20:12.894088 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a3f160a64244348a57733bad8516de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:12.894593 30873 tablet_bootstrap.cc:654] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:12.894968 30873 log.cc:824] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:12.894989 30640 tablet_service.cc:1467] Processing CreateTablet for tablet 11a3f160a64244348a57733bad8516de (DEFAULT_TABLE table=test-table [id=007ea502c0634d569a9de08055436da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:12.895308 30640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a3f160a64244348a57733bad8516de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:12.897419 30875 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Bootstrap starting.
I20260927 03:20:12.898231 30875 tablet_bootstrap.cc:654] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:12.898569 30875 log.cc:824] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:12.898788 30873 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: No bootstrap required, opened a new log
I20260927 03:20:12.898922 30873 ts_tablet_manager.cc:1397] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:20:12.899331 30875 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: No bootstrap required, opened a new log
I20260927 03:20:12.899456 30875 ts_tablet_manager.cc:1397] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:12.901551 30873 raft_consensus.cc:348] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.901823 30873 raft_consensus.cc:374] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:12.901909 30873 raft_consensus.cc:729] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331ea232bae49a0ac2925ca21752861, State: Initialized, Role: FOLLOWER
I20260927 03:20:12.901837 30875 raft_consensus.cc:348] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.902069 30875 raft_consensus.cc:374] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:12.902102 30873 consensus_queue.cc:260] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.902141 30875 raft_consensus.cc:729] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedad9c8a86c41dcac1d761541fef2d3, State: Initialized, Role: FOLLOWER
I20260927 03:20:12.902324 30875 consensus_queue.cc:260] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.902372 30873 ts_tablet_manager.cc:1428] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:20:12.902472 30572 heartbeater.cc:503] Master 127.17.52.190:38033 was elected leader, sending a full tablet report...
I20260927 03:20:12.902904 30875 ts_tablet_manager.cc:1428] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:20:12.903007 30840 heartbeater.cc:503] Master 127.17.52.190:38033 was elected leader, sending a full tablet report...
I20260927 03:20:12.903929 30876 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Bootstrap starting.
I20260927 03:20:12.904603 30876 tablet_bootstrap.cc:654] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:12.904912 30876 log.cc:824] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:12.905689 30876 tablet_bootstrap.cc:492] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: No bootstrap required, opened a new log
I20260927 03:20:12.905757 30876 ts_tablet_manager.cc:1397] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:20:12.908221 30876 raft_consensus.cc:348] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.908383 30876 raft_consensus.cc:374] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:12.908414 30876 raft_consensus.cc:729] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba68c340714c4aafb25b9da0d512b639, State: Initialized, Role: FOLLOWER
I20260927 03:20:12.908594 30876 consensus_queue.cc:260] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.908893 30876 ts_tablet_manager.cc:1428] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:20:12.909034 30706 heartbeater.cc:503] Master 127.17.52.190:38033 was elected leader, sending a full tablet report...
I20260927 03:20:12.936093 30881 raft_consensus.cc:482] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:12.936358 30881 raft_consensus.cc:504] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.937510 30881 leader_election.cc:288] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2331ea232bae49a0ac2925ca21752861 (127.17.52.129:33565), aedad9c8a86c41dcac1d761541fef2d3 (127.17.52.131:34351)
W20260927 03:20:12.941263 30573 tablet.cc:2369] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:12.941324 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.937455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36197 (local address 127.17.52.131:34351)
0927 03:20:12.937492 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.937494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.937721 (+ 227us) server_negotiation.cc:407] Connection header received
0927 03:20:12.937815 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.937816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.937824 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.937851 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:12.938040 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.938770 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.940424 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.940957 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.941000 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.941009 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.941222 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.941259 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:20:12.941723 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.937292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:34351 (local address 127.17.52.130:36197)
0927 03:20:12.937659 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.937668 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:12.937746 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:12.937910 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:12.937911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:12.937912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:12.937975 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.937977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.939184 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.939184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.940327 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.940330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.941081 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.941082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.941140 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.941143 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:12.941592 (+ 449us) client_negotiation.cc:240] Negotiation successful
0927 03:20:12.941633 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":78,"threads_started":1}
I20260927 03:20:12.941751 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.937266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:34961 (local address 127.17.52.129:33565)
0927 03:20:12.937326 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.937328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.937434 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:20:12.937537 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.937538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.937547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.937574 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:12.937776 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.938517 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.940865 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.941376 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.941423 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.941430 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.941648 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.941687 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:20:12.941990 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.937043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33565 (local address 127.17.52.130:34961)
0927 03:20:12.937358 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.937370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:12.937462 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:12.937638 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:12.937638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:12.937640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:12.937710 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.937712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.938969 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.938970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.940748 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:12.940751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.941489 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.941490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:12.941565 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.941568 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:12.941866 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 03:20:12.941909 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
I20260927 03:20:12.942224 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a3f160a64244348a57733bad8516de" candidate_uuid: "ba68c340714c4aafb25b9da0d512b639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedad9c8a86c41dcac1d761541fef2d3" is_pre_election: true
I20260927 03:20:12.942373 30794 raft_consensus.cc:2454] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba68c340714c4aafb25b9da0d512b639 in term 0.
I20260927 03:20:12.942404 30526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a3f160a64244348a57733bad8516de" candidate_uuid: "ba68c340714c4aafb25b9da0d512b639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331ea232bae49a0ac2925ca21752861" is_pre_election: true
I20260927 03:20:12.942554 30526 raft_consensus.cc:2454] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba68c340714c4aafb25b9da0d512b639 in term 0.
I20260927 03:20:12.942559 30594 leader_election.cc:302] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedad9c8a86c41dcac1d761541fef2d3, ba68c340714c4aafb25b9da0d512b639; no voters:
I20260927 03:20:12.942830 30881 raft_consensus.cc:2781] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:12.942895 30881 raft_consensus.cc:482] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:12.942919 30881 raft_consensus.cc:3037] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:12.943781 30881 raft_consensus.cc:504] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.944007 30881 leader_election.cc:288] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [CANDIDATE]: Term 1 election: Requested vote from peers 2331ea232bae49a0ac2925ca21752861 (127.17.52.129:33565), aedad9c8a86c41dcac1d761541fef2d3 (127.17.52.131:34351)
I20260927 03:20:12.944135 30526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a3f160a64244348a57733bad8516de" candidate_uuid: "ba68c340714c4aafb25b9da0d512b639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331ea232bae49a0ac2925ca21752861"
I20260927 03:20:12.944180 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a3f160a64244348a57733bad8516de" candidate_uuid: "ba68c340714c4aafb25b9da0d512b639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedad9c8a86c41dcac1d761541fef2d3"
I20260927 03:20:12.944209 30526 raft_consensus.cc:3037] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:12.944245 30794 raft_consensus.cc:3037] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:12.945114 30526 raft_consensus.cc:2454] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba68c340714c4aafb25b9da0d512b639 in term 1.
I20260927 03:20:12.945125 30794 raft_consensus.cc:2454] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba68c340714c4aafb25b9da0d512b639 in term 1.
I20260927 03:20:12.945298 30594 leader_election.cc:302] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedad9c8a86c41dcac1d761541fef2d3, ba68c340714c4aafb25b9da0d512b639; no voters:
I20260927 03:20:12.945418 30881 raft_consensus.cc:2781] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:12.945568 30881 raft_consensus.cc:686] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 1 LEADER]: Becoming Leader. State: Replica: ba68c340714c4aafb25b9da0d512b639, State: Running, Role: LEADER
I20260927 03:20:12.945709 30881 consensus_queue.cc:237] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } }
I20260927 03:20:12.946486 30373 catalog_manager.cc:5495] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba68c340714c4aafb25b9da0d512b639 (127.17.52.130). New cstate: current_term: 1 leader_uuid: "ba68c340714c4aafb25b9da0d512b639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba68c340714c4aafb25b9da0d512b639" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20260927 03:20:12.960386 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.956963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.17.52.130:46381)
0927 03:20:12.957017 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:12.957019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:12.957031 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:20:12.957102 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:12.957103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:12.957124 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:12.957199 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:12.957390 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.958156 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.959297 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:12.959585 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:12.959647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:12.959855 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:12.959988 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:12.960062 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:12.960288 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 03:20:12.960308 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:20:12.962491 30794 raft_consensus.cc:1261] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 1 FOLLOWER]: Refusing update from remote peer ba68c340714c4aafb25b9da0d512b639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:12.962553 30526 raft_consensus.cc:1261] T 11a3f160a64244348a57733bad8516de P 2331ea232bae49a0ac2925ca21752861 [term 1 FOLLOWER]: Refusing update from remote peer ba68c340714c4aafb25b9da0d512b639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:12.962903 30881 consensus_queue.cc:1035] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2331ea232bae49a0ac2925ca21752861" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:12.963048 30884 consensus_queue.cc:1035] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedad9c8a86c41dcac1d761541fef2d3" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:12.966025 30887 mvcc.cc:205] Tried to move back new op lower bound from 7333802856291196928 to 7333802856226242560. Current Snapshot: MvccSnapshot[applied={T|T < 7333802856291196928}]
I20260927 03:20:12.967613 30890 mvcc.cc:205] Tried to move back new op lower bound from 7333802856291196928 to 7333802856226242560. Current Snapshot: MvccSnapshot[applied={T|T < 7333802856291196928}]
I20260927 03:20:12.968091 30889 mvcc.cc:205] Tried to move back new op lower bound from 7333802856291196928 to 7333802856226242560. Current Snapshot: MvccSnapshot[applied={T|T < 7333802856291196928}]
I20260927 03:20:12.974826 30374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:12.974926 30374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:12.975785 30374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 833206ac25654d4891d618244f3859a9: Sending DeleteTablet for 3 replicas of tablet 11a3f160a64244348a57733bad8516de
I20260927 03:20:12.976173 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet 11a3f160a64244348a57733bad8516de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:12 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:52268
I20260927 03:20:12.976349 30640 tablet_service.cc:1514] Processing DeleteTablet for tablet 11a3f160a64244348a57733bad8516de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:12 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:52232
I20260927 03:20:12.976673 30898 tablet_replica.cc:330] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: stopping tablet replica
I20260927 03:20:12.976650 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 11a3f160a64244348a57733bad8516de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:12 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:47596
I20260927 03:20:12.976775 30898 raft_consensus.cc:2229] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:20:12.976876 30898 raft_consensus.cc:2258] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:12.977360 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30412
I20260927 03:20:12.977941 30898 ts_tablet_manager.cc:1905] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:12.979666 30898 ts_tablet_manager.cc:1918] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:12.979730 30898 log.cc:1196] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/11a3f160a64244348a57733bad8516de
I20260927 03:20:12.979992 30898 ts_tablet_manager.cc:1939] T 11a3f160a64244348a57733bad8516de P ba68c340714c4aafb25b9da0d512b639: Deleting consensus metadata
I20260927 03:20:12.980472 30359 catalog_manager.cc:4841] TS ba68c340714c4aafb25b9da0d512b639 (127.17.52.130:46381): tablet 11a3f160a64244348a57733bad8516de (table test-table [id=007ea502c0634d569a9de08055436da0]) successfully deleted
I20260927 03:20:12.986249 30899 tablet_replica.cc:330] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: stopping tablet replica
I20260927 03:20:12.987077 30899 raft_consensus.cc:2229] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:12.987659 30899 raft_consensus.cc:2258] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:12.991079 30899 ts_tablet_manager.cc:1905] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:12.992966 30899 ts_tablet_manager.cc:1918] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:12.993085 30899 log.cc:1196] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/11a3f160a64244348a57733bad8516de
I20260927 03:20:12.993415 30899 ts_tablet_manager.cc:1939] T 11a3f160a64244348a57733bad8516de P aedad9c8a86c41dcac1d761541fef2d3: Deleting consensus metadata
I20260927 03:20:12.993933 30359 catalog_manager.cc:4841] TS aedad9c8a86c41dcac1d761541fef2d3 (127.17.52.131:34351): tablet 11a3f160a64244348a57733bad8516de (table test-table [id=007ea502c0634d569a9de08055436da0]) successfully deleted
W20260927 03:20:12.998173 30361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:33565 (error 108)
I20260927 03:20:12.999249 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:12.999099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:33565 (local address 127.0.0.1:52284)
0927 03:20:12.999163 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:12.999202 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:20:12.999501 30361 catalog_manager.cc:4584] TS 2331ea232bae49a0ac2925ca21752861 (127.17.52.129:33565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11a3f160a64244348a57733bad8516de: Network error: Client connection negotiation failed: client connection to 127.17.52.129:33565: connect: Connection refused (error 111)
Sep 27 03:20:12 dist-test-slave-7j43 krb5kdc[29378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479212, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for HTTP/127.17.52.148@KRBTEST.COM
I20260927 03:20:13.000370 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30576
I20260927 03:20:13.007131 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30709
I20260927 03:20:13.012794 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30341
I20260927 03:20:13.021814 17618 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:20:13.366412 17618 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:20:13.366526 17618 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:20:13.710927 17618 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:20:13.711 UTC [29408] LOG: received smart shutdown request
2026-09-27 03:20:13.714 UTC [29408] LOG: background worker "logical replication launcher" (PID 29417) exited with exit code 1
2026-09-27 03:20:13.714 UTC [29412] LOG: shutting down
2026-09-27 03:20:13.856 UTC [29408] LOG: database system is shut down
2026-09-27T03:20:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46571 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:20:14 dist-test-slave-7j43 krb5kdc[30910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:20:14 dist-test-slave-7j43 krb5kdc[30910](info): set up 2 sockets
Sep 27 03:20:14 dist-test-slave-7j43 krb5kdc[30910](info): commencing operation
krb5kdc: starting...
W20260927 03:20:15.001060 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:20:15.119120 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44623
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:15.202003 30923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:15.202234 30923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:15.202266 30923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:15.203361 30923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:15.203428 30923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:15.203452 30923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:15.203485 30923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:15.203508 30923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:15.205061 30923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:15.205331 30923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:15.205559 30923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:15.210325 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:15.210336 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:15.210343 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:15.210608 30923 server_base.cc:1034] running on GCE node
I20260927 03:20:15.210868 30923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:15.211146 30923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:15.212338 30923 hybrid_clock.cc:648] HybridClock initialized: now 1790479215212318 us; error 32 us; skew 500 ppm
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:15.217275 30923 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:15.217727 30923 webserver.cc:460] Webserver started at http://127.17.52.190:34903/ using document root <none> and password file <none>
I20260927 03:20:15.217983 30923 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:15.218052 30923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:15.218204 30923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:15.219312 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "0bc21aefe68d4a7186550a819b65a253"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "54db22d848f5d5c24583bebe43199823"
server_key_iv: "ec550e30654921cf80d927efac5c48a5"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.219671 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "0bc21aefe68d4a7186550a819b65a253"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "54db22d848f5d5c24583bebe43199823"
server_key_iv: "ec550e30654921cf80d927efac5c48a5"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.221042 30923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:20:15.221752 30938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:15.221925 30923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:15.222026 30923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "0bc21aefe68d4a7186550a819b65a253"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "54db22d848f5d5c24583bebe43199823"
server_key_iv: "ec550e30654921cf80d927efac5c48a5"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.222105 30923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:15.256340 30923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:15.258276 30923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:15.258450 30923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:15.262444 30923 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:44623
I20260927 03:20:15.262488 30990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:44623 every 8 connection(s)
I20260927 03:20:15.262857 30923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:15.263731 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:15.264519 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30923
I20260927 03:20:15.264658 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:20:15.264859 17618 external_mini_cluster.cc:1428] Setting key 7ef108f262dfffe86fa994946933b209
I20260927 03:20:15.265532 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.189@KRBTEST.COM" created.
I20260927 03:20:15.276362 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.266324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44623 (local address 127.0.0.1:34968)
0927 03:20:15.266519 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.266562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.266990 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.267284 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.267287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.267528 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.267684 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.267688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.269288 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.269289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.269977 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.269979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.270709 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.270710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.270766 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.271311 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.271319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.272916 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.274594 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.274601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.274603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.274910 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.275202 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.275203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.275203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.275294 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.275706 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.275711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.275785 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.275893 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.276139 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":98,"thread_start_us":88,"threads_started":1}
I20260927 03:20:15.276364 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.266331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.17.52.190:44623)
0927 03:20:15.266517 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.266528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.266631 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:20:15.267143 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.267145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.267167 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.267268 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.267780 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.269195 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.270066 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.270620 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.270733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.273001 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.273015 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.273015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.273021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.274507 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.275002 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.275005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.275005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.275113 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.275379 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.275380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.275380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.275618 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.275730 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.275920 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.276068 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"spinlock_wait_cycles":7936,"thread_start_us":90,"threads_started":1}
I20260927 03:20:15.277360 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.277691 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.277581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59126)
0927 03:20:15.277627 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.277668 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:20:15.277825 30940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
I20260927 03:20:15.278239 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.278147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59132)
0927 03:20:15.278190 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.278220 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:20:15.278352 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37485: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
I20260927 03:20:15.289335 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:37485
--webserver_interface=127.17.52.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:15.327548 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } attempt: 1
I20260927 03:20:15.328073 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.327933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59142)
0927 03:20:15.328001 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.328044 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:20:15.328446 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.328365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59148)
0927 03:20:15.328407 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.328430 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:20:15.328547 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37485: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
W20260927 03:20:15.370306 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:15.370508 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:15.370532 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:15.371619 30998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:15.371654 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:15.371668 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:15.371681 30998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:15.371693 30998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:15.373200 30998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:37485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.52.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:15.373441 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:15.373639 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:15.378319 31006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:15.378355 31004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:15.378481 30998 server_base.cc:1034] running on GCE node
W20260927 03:20:15.378319 31003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:15.378808 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:15.379179 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:15.380378 30998 hybrid_clock.cc:648] HybridClock initialized: now 1790479215380353 us; error 34 us; skew 500 ppm
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:15.383029 30998 init.cc:377] Logged in from keytab as kudu/127.17.52.189@KRBTEST.COM (short username kudu)
I20260927 03:20:15.383495 30998 webserver.cc:460] Webserver started at http://127.17.52.189:37109/ using document root <none> and password file <none>
I20260927 03:20:15.383800 30998 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:15.383881 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:15.384047 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:15.385283 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data/instance:
uuid: "e83f4b674c6d4928a633d8a1352f928f"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "8fd62948c80558025ac5e7fc6ce3661b"
server_key_iv: "e1f15a7d6654a8a8e498232147e09d21"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.385689 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal/instance:
uuid: "e83f4b674c6d4928a633d8a1352f928f"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "8fd62948c80558025ac5e7fc6ce3661b"
server_key_iv: "e1f15a7d6654a8a8e498232147e09d21"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.387113 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:20:15.387965 31013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:15.388170 30998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:15.388295 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "e83f4b674c6d4928a633d8a1352f928f"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "8fd62948c80558025ac5e7fc6ce3661b"
server_key_iv: "e1f15a7d6654a8a8e498232147e09d21"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.388379 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:15.396704 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } attempt: 2
I20260927 03:20:15.397162 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.397007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59152)
0927 03:20:15.397090 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.397130 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 03:20:15.397538 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.397441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59162)
0927 03:20:15.397486 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.397517 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:20:15.397622 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37485: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37485: connect: Connection refused (error 111)
I20260927 03:20:15.400802 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:15.402667 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:15.402835 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:15.407066 30998 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.189:37485
I20260927 03:20:15.407122 31065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.189:37485 every 8 connection(s)
I20260927 03:20:15.407510 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
I20260927 03:20:15.408478 31066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:15.410178 31066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.414057 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 30998
I20260927 03:20:15.414139 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-1/wal/instance
I20260927 03:20:15.414371 17618 external_mini_cluster.cc:1428] Setting key a5fc0362e22f722870efcdd646c94c31
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:15.420076 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.410884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.17.52.190:44623)
0927 03:20:15.410942 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.410944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.411133 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:20:15.411657 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.411658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.411674 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.411733 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.412346 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.412937 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.413923 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.414167 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.414254 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.417113 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.417115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.417115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.417122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.418568 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.419065 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.419066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.419066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.419114 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.419406 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.419407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.419407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.419505 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.419554 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.419947 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.420012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:20:15.420399 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.410848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44623 (local address 127.0.0.1:34980)
0927 03:20:15.411017 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.411052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.411542 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.411801 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.411805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.412055 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.412241 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.412245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.413056 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.413059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.413832 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.413835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.414314 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.414315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.414489 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.415001 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.415011 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.417004 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.418659 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.418667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.418669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.418989 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.419181 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.419182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.419182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.419338 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.419570 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.419578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.419767 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.419926 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.420109 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260927 03:20:15.420991 31066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.52.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.188@KRBTEST.COM" created.
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:20:15.428911 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.421324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.17.52.189:37485)
0927 03:20:15.421486 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.421490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.421505 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:20:15.421521 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.421522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.421532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.421611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.421867 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.422528 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.423216 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.423560 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.423642 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.425663 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.425671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.425672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.425679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.427553 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.427978 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.427981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.427982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.428042 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.428260 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.428261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.428261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.428406 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.428477 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.428708 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.428819 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260927 03:20:15.428921 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.421255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59164)
0927 03:20:15.421324 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.421334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.421373 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.421610 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.421611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.421750 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.421792 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.421794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.422613 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.422614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.423121 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.423122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.423637 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.423638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.423705 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.424083 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.424087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.425558 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.427661 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.427661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.427662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.427880 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.428120 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.428122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.428122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.428195 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.428475 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.428476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.428529 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.428619 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.428752 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:20:15.429785 31066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.430190 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.430042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54420)
0927 03:20:15.430098 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.430155 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:20:15.430359 31017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
I20260927 03:20:15.430853 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.430736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54436)
0927 03:20:15.430786 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.430821 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:20:15.430987 31066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:44549: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
I20260927 03:20:15.441437 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:44549
--webserver_interface=127.17.52.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:15.480234 31066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } attempt: 1
I20260927 03:20:15.480834 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.480659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54446)
0927 03:20:15.480720 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.480772 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:20:15.481223 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.481127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54452)
0927 03:20:15.481175 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.481204 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:20:15.481323 31066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:44549: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
I20260927 03:20:15.488759 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } attempt: 3
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:20:15.496166 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.489126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.17.52.189:37485)
0927 03:20:15.489164 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.489166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.489182 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:20:15.489243 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.489244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.489254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.489309 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.489775 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.490273 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.491260 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.491522 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.491576 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.493575 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.493577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.493577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.493585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.494983 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.495383 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.495385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.495386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.495445 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.495640 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.495641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.495642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.495754 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.495844 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.495997 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.496077 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:20:15.496182 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.489074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59170)
0927 03:20:15.489129 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.489140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.489181 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.489331 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.489332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.489534 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.489610 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.489613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.490348 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.490350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.491144 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.491147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.491595 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.491596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.491683 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.492163 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.492168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.493487 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.495114 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.495115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.495116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.495291 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.495510 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.495511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.495511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.495576 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.495837 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.495837 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.495886 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.495973 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.496078 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:20:15.496548 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.496932 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.496845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54464)
0927 03:20:15.496886 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.496914 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:20:15.497247 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.497162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54474)
0927 03:20:15.497204 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.497227 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:20:15.497344 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:44549: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
W20260927 03:20:15.523157 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:15.523373 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:15.523411 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:15.524519 31073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:15.524582 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:15.524639 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:15.524682 31073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:15.524713 31073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:15.526260 31073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44623,127.17.52.189:37485,127.17.52.188:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.52.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:15.526518 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:15.526736 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:20:15.528483 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } attempt: 1
I20260927 03:20:15.528889 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.528764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54480)
0927 03:20:15.528824 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.528861 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:20:15.529233 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.529152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54496)
0927 03:20:15.529194 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.529218 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:20:15.529336 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:44549: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
W20260927 03:20:15.531544 31081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:15.531594 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:15.531746 31073 server_base.cc:1034] running on GCE node
W20260927 03:20:15.531832 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:15.532064 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:15.532348 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:15.533505 31073 hybrid_clock.cc:648] HybridClock initialized: now 1790479215533492 us; error 23 us; skew 500 ppm
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:15.535836 31073 init.cc:377] Logged in from keytab as kudu/127.17.52.188@KRBTEST.COM (short username kudu)
I20260927 03:20:15.536230 31073 webserver.cc:460] Webserver started at http://127.17.52.188:45549/ using document root <none> and password file <none>
I20260927 03:20:15.536465 31073 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:15.536533 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:15.536715 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:15.537724 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data/instance:
uuid: "690cd9c80f514d8285072ea8540aba44"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "49505f5fada4616f3acbe058ec468dd5"
server_key_iv: "e7cef83f7bb06452552c215e8d73e2f9"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.538071 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal/instance:
uuid: "690cd9c80f514d8285072ea8540aba44"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "49505f5fada4616f3acbe058ec468dd5"
server_key_iv: "e7cef83f7bb06452552c215e8d73e2f9"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.539332 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:20:15.539999 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:15.540159 31073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:15.540256 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "690cd9c80f514d8285072ea8540aba44"
format_stamp: "Formatted at 2026-09-27 03:20:15 on dist-test-slave-7j43"
server_key: "49505f5fada4616f3acbe058ec468dd5"
server_key_iv: "e7cef83f7bb06452552c215e8d73e2f9"
server_key_version: "encryptionkey@0"
I20260927 03:20:15.540346 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:15.549464 31066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } attempt: 2
I20260927 03:20:15.549906 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.549771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54512)
0927 03:20:15.549835 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.549874 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:20:15.550257 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.550163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54528)
0927 03:20:15.550206 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.550233 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:20:15.550398 31066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:44549: Network error: Client connection negotiation failed: client connection to 127.17.52.188:44549: connect: Connection refused (error 111)
I20260927 03:20:15.566170 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:15.568022 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:15.568188 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:15.572357 31073 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.188:44549
I20260927 03:20:15.572394 31140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.188:44549 every 8 connection(s)
I20260927 03:20:15.572813 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
I20260927 03:20:15.573657 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:15.575196 31141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.576225 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31073
I20260927 03:20:15.576337 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479066103559-17618-0/minicluster-data/master-2/wal/instance
I20260927 03:20:15.576510 17618 external_mini_cluster.cc:1428] Setting key 637a7575878e4b4510e1ca72c66ca7ff
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:20:15.585372 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.575909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.17.52.190:44623)
0927 03:20:15.575967 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.575969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.576153 (+ 184us) server_negotiation.cc:407] Connection header received
0927 03:20:15.576752 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.576753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.576763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.576851 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.577585 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.578176 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.579126 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.579311 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.579362 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.581725 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.581726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.581726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.581732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.583266 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584407 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.584408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.584408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.584487 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584755 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.584756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.584756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.584892 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.584962 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.585186 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.585274 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:20:15.585561 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.575872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:44623 (local address 127.0.0.1:34996)
0927 03:20:15.576028 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.576068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.576646 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.576942 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.576947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.577237 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.577492 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.577499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.578253 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.578254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.579046 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.579048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.579396 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.579397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.579540 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.580084 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.580092 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.581633 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.584084 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.584090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.584335 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.584554 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.584555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.584691 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.584960 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.584964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.585031 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.585127 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.585283 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 03:20:15.585702 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.576904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.17.52.188:44549)
0927 03:20:15.577213 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.577216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.577230 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:20:15.577246 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.577248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.577267 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.577323 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.577970 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.578869 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.579768 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.580168 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.580220 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.582492 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.582499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.582500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.582506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.583982 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584407 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.584410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.584411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.584526 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.584754 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.584755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.584755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.585040 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.585254 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.585308 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.585375 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":287,"thread_start_us":109,"threads_started":1}
I20260927 03:20:15.586195 31141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:20:15.594931 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.586485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.17.52.189:37485)
0927 03:20:15.586527 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.586529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.586598 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:20:15.586656 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.586657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.586668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.586710 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.587002 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.587439 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.588448 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.588697 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.588763 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.590810 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.590812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.590812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.590819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.593652 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.594061 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.594063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.594063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.594130 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.594346 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.594347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.594348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.594482 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.594524 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.594743 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.594833 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:20:15.595216 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.586413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37485 (local address 127.0.0.1:59176)
0927 03:20:15.586488 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.586544 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.586595 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.586714 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.586715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.586875 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.586933 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.586934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.587509 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.587510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.588367 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.588369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.588848 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.588849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.588930 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.589439 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.589444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.590722 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.593745 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.593746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.593747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.593965 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.594209 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.594210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.594210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.594271 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.594574 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.594575 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.594620 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.595011 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.595103 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:15.595599 31141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:20:15.598655 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.589044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.17.52.190:44623)
0927 03:20:15.589408 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.589410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.589418 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:20:15.589438 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.589439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.589467 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.589521 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.590148 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.590786 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.591761 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.592057 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.592114 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.596037 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.596039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.596039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.596045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.597509 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.597884 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.597885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.597885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.597941 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.598160 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.598161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.598161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.598293 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.598487 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.598527 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.598596 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:20:15.602481 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.595882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54540)
0927 03:20:15.595919 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.595928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.595961 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.596126 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.596127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.596281 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.596319 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.596320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.597140 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.597141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.597654 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.597656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.598067 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.598068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.598128 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.598494 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.598500 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.599828 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.601357 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.601358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.601358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.601514 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.601878 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.601879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.601879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.601939 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.602176 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.602177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.602214 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.602280 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.602382 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:20:15.602537 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.595945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.17.52.188:44549)
0927 03:20:15.595997 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.595999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.596042 (+ 43us) server_negotiation.cc:407] Connection header received
0927 03:20:15.596058 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.596059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.596066 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.596112 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.596426 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.597042 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.597738 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.597998 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.598054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.599905 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.599906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.599906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.599912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.601254 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.601596 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.601598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.601598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.601650 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.602012 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.602013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.602013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.602110 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.602184 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.602306 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.602430 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":17}
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:20:15.604517 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } attempt: 2
I20260927 03:20:15.605026 31141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: Bootstrap starting.
I20260927 03:20:15.606088 31141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:15.606298 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.599307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.17.52.189:37485)
0927 03:20:15.599341 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.599343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.599385 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:20:15.599454 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.599455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.599465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.599518 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.600068 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.600504 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.601153 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.601344 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.601427 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.603619 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.603620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.603620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.603627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.605145 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.605494 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.605495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.605496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.605568 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.605782 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.605782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.605783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.605903 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.606084 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.606126 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.606196 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:20:15.606402 31141 log.cc:824] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:15.607211 31141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: No bootstrap required, opened a new log
I20260927 03:20:15.608101 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:20:15.609550 31141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.609792 31141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:15.609900 31141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690cd9c80f514d8285072ea8540aba44, State: Initialized, Role: FOLLOWER
I20260927 03:20:15.610054 31141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.610479 31141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:15.612046 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.604801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54542)
0927 03:20:15.604836 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.604847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.604887 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.605006 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.605007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.605192 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.605233 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.605235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.606040 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.606041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.606751 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.606754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.607174 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.607175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.607229 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.607658 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.607664 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.609020 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.610894 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.610895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.610895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.611133 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.611369 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.611370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.611370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.611428 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.611750 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.611751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.611798 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.611850 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.611951 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:20:15.612557 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.604856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.17.52.188:44549)
0927 03:20:15.604889 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.604890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.604898 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:20:15.604940 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.604941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.604949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.605001 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.605334 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.605932 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.606840 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.607111 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.607176 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.609113 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.609114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.609114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.609121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.610800 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.611224 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.611224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.611225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.611283 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.611496 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.611496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.611497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.611663 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.612354 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.612397 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.612471 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:20:15.613005 31166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:20:15.613057 31166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:20:15.614434 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: Bootstrap starting.
I20260927 03:20:15.615160 30991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:15.615494 30991 log.cc:824] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:15.612918 31073 master_runner.cc:186] Error getting master registration for 127.17.52.190:44623: OK, instance_id { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" instance_seqno: 1790479215261282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:20:15.615902 31073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:20:15.616389 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: No bootstrap required, opened a new log
I20260927 03:20:15.618474 30991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.618667 30991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:15.618750 30991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc21aefe68d4a7186550a819b65a253, State: Initialized, Role: FOLLOWER
I20260927 03:20:15.618940 30991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.619292 30991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:15.622601 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.610176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.17.52.189:37485)
0927 03:20:15.610217 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.610218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.610385 (+ 167us) server_negotiation.cc:407] Connection header received
0927 03:20:15.610479 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.610480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.610489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.610518 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.611153 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.611886 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.616417 (+ 4531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.616737 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.616775 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.618322 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.618323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.618323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.618330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.620404 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.620840 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.620842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.620842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.620894 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.621104 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.621105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.621105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.621228 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.621788 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.622431 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.622511 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:20:15.621747 30923 master_runner.cc:186] Error getting master registration for 127.17.52.189:37485: OK, instance_id { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" instance_seqno: 1790479215405909 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:20:15.624431 30923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:20:15.624936 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.609513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.17.52.188:44549)
0927 03:20:15.609945 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.609946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.610084 (+ 138us) server_negotiation.cc:407] Connection header received
0927 03:20:15.610102 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.610103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.610112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.610179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.612417 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.616113 (+ 3696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.617804 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.618161 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.618224 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.621048 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.621050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.621050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.621056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.623459 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.623987 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.623988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.623988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.624035 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.624276 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.624277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.624277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.624400 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.624477 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.624701 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.624789 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":89,"threads_started":1}
I20260927 03:20:15.625046 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.609765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.17.52.190:44623)
0927 03:20:15.609855 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.609856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.609867 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:20:15.609889 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.609890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.609900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.610275 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.611283 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.611961 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.618405 (+ 6444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.618718 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.618759 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.621350 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.621351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.621352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.621358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.622897 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.623980 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.623981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.623982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.624035 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.624258 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.624259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.624259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.624387 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.624482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.624890 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.624973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 03:20:15.625777 31180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:20:15.625890 31180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:20:15.625917 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.626022 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.628690 31169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.628777 31169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.641551 31066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } attempt: 3
Sep 27 03:20:15 dist-test-slave-7j43 krb5kdc[30910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:20:15.648449 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.641882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.17.52.188:44549)
0927 03:20:15.641939 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:15.641941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:15.641949 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:20:15.641971 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:15.641971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:15.641983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:15.642054 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:15.642336 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.642861 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.643568 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.643811 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.643894 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.645950 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:15.645952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:15.645952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:15.645959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:15.647318 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.647669 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.647670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.647670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.647724 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:15.647932 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:15.647933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:15.647933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:15.648050 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:15.648119 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:15.648289 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:20:15.648367 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:20:15.648449 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:15.641811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:44549 (local address 127.0.0.1:54554)
0927 03:20:15.641859 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:15.641871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:15.641926 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:15.642057 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:15.642058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:15.642203 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:15.642239 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.642241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.642938 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.642939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.643485 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:15.643487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:15.643891 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:15.643892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:15.643956 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:15.644735 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:15.644739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:15.645855 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:15.647417 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.647418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.647418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.647588 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.647788 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:15.647789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:15.647789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:15.647858 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:15.648123 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:15.648125 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:15.648168 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:20:15.648252 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:20:15.648352 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:20:15.650242 31066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: Bootstrap starting.
I20260927 03:20:15.650897 31066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:15.651116 31066 log.cc:824] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:15.651880 31066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: No bootstrap required, opened a new log
I20260927 03:20:15.653396 31066 raft_consensus.cc:348] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.653548 31066 raft_consensus.cc:374] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:15.653623 31066 raft_consensus.cc:729] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83f4b674c6d4928a633d8a1352f928f, State: Initialized, Role: FOLLOWER
I20260927 03:20:15.653761 31066 consensus_queue.cc:260] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.654032 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.654095 31066 sys_catalog.cc:564] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:15.654160 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:20:15.655907 31194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:20:15.655988 31194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:20:15.658010 30998 master_runner.cc:263] No leader master found from master e83f4b674c6d4928a633d8a1352f928f
I20260927 03:20:15.658056 30998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:20:15.759258 30944 tablet.cc:2369] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:20:15.818856 31094 tablet.cc:2369] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:15.851784 31169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:15.851967 31169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.852247 31169 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83f4b674c6d4928a633d8a1352f928f (127.17.52.189:37485), 690cd9c80f514d8285072ea8540aba44 (127.17.52.188:44549)
I20260927 03:20:15.852586 31115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bc21aefe68d4a7186550a819b65a253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "690cd9c80f514d8285072ea8540aba44" is_pre_election: true
I20260927 03:20:15.852610 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bc21aefe68d4a7186550a819b65a253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f4b674c6d4928a633d8a1352f928f" is_pre_election: true
I20260927 03:20:15.852744 31040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc21aefe68d4a7186550a819b65a253 in term 0.
I20260927 03:20:15.852747 31115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc21aefe68d4a7186550a819b65a253 in term 0.
I20260927 03:20:15.852928 30940 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc21aefe68d4a7186550a819b65a253, e83f4b674c6d4928a633d8a1352f928f; no voters:
I20260927 03:20:15.853071 31169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:15.853173 31169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:15.853212 31169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:15.853935 31169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.854108 31169 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [CANDIDATE]: Term 1 election: Requested vote from peers e83f4b674c6d4928a633d8a1352f928f (127.17.52.189:37485), 690cd9c80f514d8285072ea8540aba44 (127.17.52.188:44549)
I20260927 03:20:15.854259 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bc21aefe68d4a7186550a819b65a253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f4b674c6d4928a633d8a1352f928f"
I20260927 03:20:15.854261 31115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bc21aefe68d4a7186550a819b65a253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "690cd9c80f514d8285072ea8540aba44"
I20260927 03:20:15.854373 31040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:15.854368 31115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:15.855131 31040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc21aefe68d4a7186550a819b65a253 in term 1.
I20260927 03:20:15.855154 31115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc21aefe68d4a7186550a819b65a253 in term 1.
I20260927 03:20:15.855311 30940 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc21aefe68d4a7186550a819b65a253, e83f4b674c6d4928a633d8a1352f928f; no voters:
I20260927 03:20:15.855432 31169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:15.855557 31169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc21aefe68d4a7186550a819b65a253, State: Running, Role: LEADER
I20260927 03:20:15.855654 31169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } }
I20260927 03:20:15.856067 31196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc21aefe68d4a7186550a819b65a253. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.856159 31196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:15.856355 31198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:15.856535 31198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:15.857842 31040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [term 1 FOLLOWER]: Refusing update from remote peer 0bc21aefe68d4a7186550a819b65a253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:15.857852 31115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [term 1 FOLLOWER]: Refusing update from remote peer 0bc21aefe68d4a7186550a819b65a253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:15.858142 31196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:15.858237 31169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:15.859341 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc21aefe68d4a7186550a819b65a253. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.859369 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc21aefe68d4a7186550a819b65a253. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.859428 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.859449 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.860297 31169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.860374 31169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:15.860690 31198 catalog_manager.cc:1347] Generated new cluster ID: e5232e4ae03640e7ba3339d4858ec3c8
I20260927 03:20:15.860765 31200 mvcc.cc:205] Tried to move back new op lower bound from 7333802868151656448 to 7333802868145393664. Current Snapshot: MvccSnapshot[applied={T|T < 7333802868151656448}]
I20260927 03:20:15.860733 31196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.860831 31196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:15.860841 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.860741 31198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:20:15.860919 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.861060 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bc21aefe68d4a7186550a819b65a253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc21aefe68d4a7186550a819b65a253" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44623 } } peers { permanent_uuid: "e83f4b674c6d4928a633d8a1352f928f" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37485 } } peers { permanent_uuid: "690cd9c80f514d8285072ea8540aba44" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 44549 } } }
I20260927 03:20:15.861135 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:20:15.878975 31198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:20:15.879606 31198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:15.895072 31198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc21aefe68d4a7186550a819b65a253: Generated new TSK 0
I20260927 03:20:15.895291 31198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:20:16.613469 31166 catalog_manager.cc:1259] Loaded cluster ID: e5232e4ae03640e7ba3339d4858ec3c8
I20260927 03:20:16.613538 31166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: loading cluster ID for follower catalog manager: success
I20260927 03:20:16.614293 31166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: acquiring CA information for follower catalog manager: success
I20260927 03:20:16.614533 31166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 690cd9c80f514d8285072ea8540aba44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:20:16.656399 31194 catalog_manager.cc:1259] Loaded cluster ID: e5232e4ae03640e7ba3339d4858ec3c8
I20260927 03:20:16.656492 31194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: loading cluster ID for follower catalog manager: success
I20260927 03:20:16.657346 31194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: acquiring CA information for follower catalog manager: success
I20260927 03:20:16.657658 31194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e83f4b674c6d4928a633d8a1352f928f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:20:16.955987 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30923
I20260927 03:20:16.962088 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 30998
I20260927 03:20:16.967165 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31073
2026-09-27T03:20:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3001 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:20:17 dist-test-slave-7j43 krb5kdc[31213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:20:17 dist-test-slave-7j43 krb5kdc[31213](info): set up 2 sockets
Sep 27 03:20:17 dist-test-slave-7j43 krb5kdc[31213](info): commencing operation
krb5kdc: starting...
W20260927 03:20:17.997123 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:20:18.112848 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33087
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:18.194202 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:18.194402 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:18.194427 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:18.195648 31226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:18.195693 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:18.195709 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:18.195747 31226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:18.195778 31226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:18.197333 31226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:18.197547 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:18.197777 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:18.202580 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:18.202575 31234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:18.202718 31232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:18.203053 31226 server_base.cc:1034] running on GCE node
I20260927 03:20:18.203292 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:18.203569 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:18.204725 31226 hybrid_clock.cc:648] HybridClock initialized: now 1790479218204720 us; error 38 us; skew 500 ppm
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479218, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:18.207218 31226 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:18.207614 31226 webserver.cc:460] Webserver started at http://127.17.52.190:41557/ using document root <none> and password file <none>
I20260927 03:20:18.207849 31226 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:18.207914 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:18.208063 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:18.209134 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "37249380768f47de89b2e80752c9a968"
format_stamp: "Formatted at 2026-09-27 03:20:18 on dist-test-slave-7j43"
server_key: "33ce23bfe0de230159db7cf2e6fe9c43"
server_key_iv: "c2d4b3b9f8b9e31c1cdcb212c7fa4c5b"
server_key_version: "encryptionkey@0"
I20260927 03:20:18.209491 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "37249380768f47de89b2e80752c9a968"
format_stamp: "Formatted at 2026-09-27 03:20:18 on dist-test-slave-7j43"
server_key: "33ce23bfe0de230159db7cf2e6fe9c43"
server_key_iv: "c2d4b3b9f8b9e31c1cdcb212c7fa4c5b"
server_key_version: "encryptionkey@0"
I20260927 03:20:18.210753 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:20:18.211455 31241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:18.211611 31226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:18.211712 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "37249380768f47de89b2e80752c9a968"
format_stamp: "Formatted at 2026-09-27 03:20:18 on dist-test-slave-7j43"
server_key: "33ce23bfe0de230159db7cf2e6fe9c43"
server_key_iv: "c2d4b3b9f8b9e31c1cdcb212c7fa4c5b"
server_key_version: "encryptionkey@0"
I20260927 03:20:18.211782 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:18.222474 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:18.224385 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:18.224553 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:18.228380 31226 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33087
I20260927 03:20:18.228412 31293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33087 every 8 connection(s)
I20260927 03:20:18.228817 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:18.229658 31294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:18.231957 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: Bootstrap starting.
I20260927 03:20:18.232532 31294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:18.232859 31294 log.cc:824] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:18.233542 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: No bootstrap required, opened a new log
I20260927 03:20:18.235143 31294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } }
I20260927 03:20:18.235284 31294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:18.235308 31294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37249380768f47de89b2e80752c9a968, State: Initialized, Role: FOLLOWER
I20260927 03:20:18.235459 31294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } }
I20260927 03:20:18.235534 31294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:18.235556 31294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:18.235610 31294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:18.236171 31294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } }
I20260927 03:20:18.236258 31294 leader_election.cc:302] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37249380768f47de89b2e80752c9a968; no voters:
I20260927 03:20:18.236443 31294 leader_election.cc:288] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:20:18.236545 31297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:18.236737 31294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:18.236801 31297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [term 1 LEADER]: Becoming Leader. State: Replica: 37249380768f47de89b2e80752c9a968, State: Running, Role: LEADER
I20260927 03:20:18.237118 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31226
I20260927 03:20:18.237224 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:20:18.237413 17618 external_mini_cluster.cc:1428] Setting key 19e40995caf4092b73f156d8ccd4b669
I20260927 03:20:18.236923 31297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } }
I20260927 03:20:18.238169 31298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37249380768f47de89b2e80752c9a968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } } }
I20260927 03:20:18.238346 31299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37249380768f47de89b2e80752c9a968. Latest consensus state: current_term: 1 leader_uuid: "37249380768f47de89b2e80752c9a968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37249380768f47de89b2e80752c9a968" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33087 } } }
I20260927 03:20:18.238515 31299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:18.238500 31298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968 [sys.catalog]: This master's current role is: LEADER
W20260927 03:20:18.238729 31312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:20:18.238827 31312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:20:18.239014 31315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:18.239274 31315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:18.240857 31315 catalog_manager.cc:1347] Generated new cluster ID: bb02fd49148946eb8ecd6926b190377b
I20260927 03:20:18.240911 31315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:18.249308 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:18.238092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.17.52.190:33087)
0927 03:20:18.238418 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:18.238429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:18.238466 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:20:18.238591 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:18.238600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:18.238665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:18.238822 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:18.239607 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:18.241019 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:18.242065 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:18.242723 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:18.242847 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:18.245606 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:18.245611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:18.245611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:18.245623 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:18.247595 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:18.248016 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:18.248019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:18.248019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:18.248140 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:18.248362 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:18.248362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:18.248363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:18.248678 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:18.248827 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:18.248962 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:20:18.249092 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":70,"threads_started":1}
I20260927 03:20:18.250732 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:20:18.265512 31315 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:20:18.266276 31315 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:18.276621 31315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37249380768f47de89b2e80752c9a968: Generated new TSK 0
I20260927 03:20:18.276836 31315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:20:18.304514 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31226
2026-09-27T03:20:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1336 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31321](info): set up 2 sockets
Sep 27 03:20:18 dist-test-slave-7j43 krb5kdc[31321](info): commencing operation
krb5kdc: starting...
W20260927 03:20:19.337090 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:20:19.447723 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:19.528903 31334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:19.529148 31334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:19.529191 31334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:19.530375 31334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:19.530449 31334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:19.530476 31334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:19.530501 31334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:19.530535 31334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:19.532011 31334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:19.532300 31334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:19.532536 31334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:19.537395 31339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:19.537407 31340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:19.537443 31342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:19.537619 31334 server_base.cc:1034] running on GCE node
I20260927 03:20:19.537916 31334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:19.538144 31334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:19.539320 31334 hybrid_clock.cc:648] HybridClock initialized: now 1790479219539312 us; error 29 us; skew 500 ppm
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:19.541775 31334 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:19.542137 31334 webserver.cc:460] Webserver started at http://127.17.52.190:42739/ using document root <none> and password file <none>
I20260927 03:20:19.542326 31334 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:19.542366 31334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:19.542449 31334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:19.543401 31334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "6206ce6730234cdfa9b82d8a5b72e890"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "3bac34e8e0198d13cd143eb7bd9fcc3e"
server_key_iv: "46101d0edbe9cee6b037ad2bc14ce683"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.543694 31334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "6206ce6730234cdfa9b82d8a5b72e890"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "3bac34e8e0198d13cd143eb7bd9fcc3e"
server_key_iv: "46101d0edbe9cee6b037ad2bc14ce683"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.544934 31334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:20:19.545604 31349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.545797 31334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.545892 31334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "6206ce6730234cdfa9b82d8a5b72e890"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "3bac34e8e0198d13cd143eb7bd9fcc3e"
server_key_iv: "46101d0edbe9cee6b037ad2bc14ce683"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.545965 31334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:19.561587 31334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:19.563534 31334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:19.563647 31334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:19.567273 31334 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33681
I20260927 03:20:19.567337 31401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33681 every 8 connection(s)
I20260927 03:20:19.567652 31334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:19.568650 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:19.570907 31402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap starting.
I20260927 03:20:19.571457 31402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:19.571734 31402 log.cc:824] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:19.572201 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31334
I20260927 03:20:19.572309 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:20:19.572489 17618 external_mini_cluster.cc:1428] Setting key 11861ec2ca33a739e73e149d97b5e614
I20260927 03:20:19.572521 31402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: No bootstrap required, opened a new log
I20260927 03:20:19.574755 31402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:19.574939 31402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:19.574978 31402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Initialized, Role: FOLLOWER
I20260927 03:20:19.575124 31402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:19.575196 31402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:19.575224 31402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:19.575268 31402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:19.575927 31402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:19.576030 31402 leader_election.cc:302] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6206ce6730234cdfa9b82d8a5b72e890; no voters:
I20260927 03:20:19.576176 31402 leader_election.cc:288] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:20:19.576220 31407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:19.576398 31407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 LEADER]: Becoming Leader. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Running, Role: LEADER
I20260927 03:20:19.576400 31402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:19.576560 31407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:19.576925 31407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6206ce6730234cdfa9b82d8a5b72e890. Latest consensus state: current_term: 1 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:19.576998 31407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:19.577183 31414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:19.577674 31414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:19.577773 31408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:19.577898 31408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:19.579955 31414 catalog_manager.cc:1347] Generated new cluster ID: a34a36b174584577b7176d3e0675bab0
I20260927 03:20:19.580014 31414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:19.584890 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:19.572996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.17.52.190:33681)
0927 03:20:19.573199 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:19.573209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:19.573245 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:20:19.573941 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:19.573949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:19.574011 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:19.574158 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:19.574707 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.576370 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.577609 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.578278 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.578387 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:19.580986 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:19.580991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:19.580991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:19.581003 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:19.582970 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.583350 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.583353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.583354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.583459 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.583653 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.583654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.583654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.583969 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:19.584127 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:19.584204 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:20:19.584445 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260927 03:20:19.587000 31414 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:20:19.587483 31414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:19.596027 31414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Generated new TSK 0
I20260927 03:20:19.596199 31414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:20:19.625233 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:19.721540 31428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:19.721731 31428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:19.721755 31428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:19.722878 31428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:19.722919 31428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:19.722945 31428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:19.724360 31428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:19.724615 31428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:19.724860 31428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:19.730646 31436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:19.730782 31428 server_base.cc:1034] running on GCE node
W20260927 03:20:19.730674 31434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:19.730665 31433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:19.731122 31428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:19.731395 31428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:19.732551 31428 hybrid_clock.cc:648] HybridClock initialized: now 1790479219732533 us; error 35 us; skew 500 ppm
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:19.735280 31428 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:19.735760 31428 webserver.cc:460] Webserver started at http://127.17.52.129:43237/ using document root <none> and password file <none>
I20260927 03:20:19.736014 31428 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:19.736066 31428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:19.736186 31428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:19.737473 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "9aacd72f389f243ae2a2df239fc3db90"
server_key_iv: "1e65ac64bf6e092468b96a3d99c636bc"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.737898 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "9aacd72f389f243ae2a2df239fc3db90"
server_key_iv: "1e65ac64bf6e092468b96a3d99c636bc"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.739455 31428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:20:19.740218 31443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.740418 31428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:19.740500 31428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "9aacd72f389f243ae2a2df239fc3db90"
server_key_iv: "1e65ac64bf6e092468b96a3d99c636bc"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.740562 31428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:19.749756 31428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:19.751848 31428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:19.751951 31428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:19.752105 31428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:19.752400 31428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:19.752434 31428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.752455 31428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:19.752503 31428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.758052 31428 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:36979
I20260927 03:20:19.758102 31556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:36979 every 8 connection(s)
I20260927 03:20:19.758380 31428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:19.760215 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31428
I20260927 03:20:19.760336 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:20:19.760520 17618 external_mini_cluster.cc:1428] Setting key b086fd0512b50e10c888f509b5e9f1ba
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:19.767678 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:19.759028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59065 (local address 127.17.52.190:33681)
0927 03:20:19.759067 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:19.759069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:19.759257 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:20:19.759693 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:19.759694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:19.759703 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:19.759758 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:19.760733 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.761363 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.762357 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.762578 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.762654 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:19.764927 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:19.764929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:19.764929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:19.764937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:19.766343 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.766749 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.766750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.766751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.766812 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.767052 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.767053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.767053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.767169 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:19.767241 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:19.767501 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:20:19.767586 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:20:19.767865 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:19.758990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:59065)
0927 03:20:19.759136 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:19.759174 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:19.759619 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:19.759807 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:19.759811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:19.760349 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:19.760565 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:19.760642 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.761459 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.761460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:19.762247 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:19.762250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.762664 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.762665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:19.762788 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:19.763257 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:19.763267 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:19.764839 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:19.766442 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:19.766449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:19.766451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:19.766663 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:19.766877 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:19.766878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:19.766879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:19.766986 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:19.767246 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:19.767252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:19.767379 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:20:19.767481 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:20:19.767631 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
I20260927 03:20:19.768293 31557 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:19.768393 31557 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:19.768607 31557 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:19.769153 31366 ts_manager.cc:194] Registered new tserver with Master: 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979)
I20260927 03:20:19.769834 31366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:59065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:20:19.783315 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:19.864441 31562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:19.864708 31562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:19.864737 31562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:19.865887 31562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:19.865928 31562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:19.865954 31562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:19.867439 31562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:19.867666 31562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:19.867894 31562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:19.872635 31567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:19.872614 31570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:19.872668 31568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:19.872857 31562 server_base.cc:1034] running on GCE node
I20260927 03:20:19.873147 31562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:19.873340 31562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:19.874511 31562 hybrid_clock.cc:648] HybridClock initialized: now 1790479219874492 us; error 35 us; skew 500 ppm
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:19.877034 31562 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:19.877507 31562 webserver.cc:460] Webserver started at http://127.17.52.130:41425/ using document root <none> and password file <none>
I20260927 03:20:19.877768 31562 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:19.877838 31562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:19.877991 31562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:19.879050 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "308c50f6f74d313ccd926334f3918c6d"
server_key_iv: "c76ac045823b6465793cf4408d00033d"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.879403 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "308c50f6f74d313ccd926334f3918c6d"
server_key_iv: "c76ac045823b6465793cf4408d00033d"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.880703 31562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:20:19.881410 31577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.881582 31562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:19.881675 31562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "308c50f6f74d313ccd926334f3918c6d"
server_key_iv: "c76ac045823b6465793cf4408d00033d"
server_key_version: "encryptionkey@0"
I20260927 03:20:19.881755 31562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:19.898012 31562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:19.900164 31562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:19.900323 31562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:19.900537 31562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:19.900848 31562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:19.900902 31562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.900938 31562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:19.900970 31562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:19.906616 31562 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:37637
I20260927 03:20:19.906666 31690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:37637 every 8 connection(s)
I20260927 03:20:19.906942 31562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:20:19.907909 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31562
I20260927 03:20:19.908015 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:20:19.908211 17618 external_mini_cluster.cc:1428] Setting key 1aa67adcdd671b16e7b8491ed9bba647
Sep 27 03:20:19 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479219, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
I20260927 03:20:19.917774 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:19.907567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:43281 (local address 127.17.52.190:33681)
0927 03:20:19.907631 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:19.907633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:19.907820 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:20:19.908504 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:19.908505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:19.908517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:19.908599 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:19.909609 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.910420 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.911858 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.912224 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.912309 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:19.914676 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:19.914678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:19.914679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:19.914688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:19.916320 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.916739 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.916740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.916741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.916808 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:19.917063 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:19.917064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:19.917065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:19.917226 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:19.917302 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:19.917562 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:20:19.917651 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:20:19.917903 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:19.907560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:43281)
0927 03:20:19.907700 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:19.907732 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:19.908373 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:19.908677 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:19.908681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:19.909323 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:19.909511 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:19.909516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.910575 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.910577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:19.911749 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:19.911752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:19.912312 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:19.912313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:19.912409 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:19.912966 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:19.912974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:19.914580 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:19.916405 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:19.916410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:19.916412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:19.916667 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:19.916889 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:19.916890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:19.916890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:19.916981 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:19.917313 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:19.917317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:19.917420 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 03:20:19.917489 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:20:19.917601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260927 03:20:19.918426 31691 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:19.918555 31691 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:19.918738 31691 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:19.919066 31366 ts_manager.cc:194] Registered new tserver with Master: 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637)
I20260927 03:20:19.919441 31366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:43281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:20:19.932991 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:20.014549 31696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:20.014763 31696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:20.014810 31696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:20.015941 31696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:20.016004 31696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:20.016052 31696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:20.017529 31696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:20.017783 31696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:20.018008 31696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:20.022733 31701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.022738 31702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.022744 31704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.022964 31696 server_base.cc:1034] running on GCE node
I20260927 03:20:20.023162 31696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:20.023355 31696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:20.024492 31696 hybrid_clock.cc:648] HybridClock initialized: now 1790479220024477 us; error 29 us; skew 500 ppm
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:20.026861 31696 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:20.027263 31696 webserver.cc:460] Webserver started at http://127.17.52.131:42377/ using document root <none> and password file <none>
I20260927 03:20:20.027488 31696 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:20.027554 31696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:20.027699 31696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:20.028744 31696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "7918afab54d542da9394952f38842aa9"
format_stamp: "Formatted at 2026-09-27 03:20:20 on dist-test-slave-7j43"
server_key: "6e2e6548670bc4101a5bbe3b62193316"
server_key_iv: "dd42d6877b671c88728446f58cb1a36e"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.029112 31696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "7918afab54d542da9394952f38842aa9"
format_stamp: "Formatted at 2026-09-27 03:20:20 on dist-test-slave-7j43"
server_key: "6e2e6548670bc4101a5bbe3b62193316"
server_key_iv: "dd42d6877b671c88728446f58cb1a36e"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.030361 31696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:20:20.031044 31711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.031201 31696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:20:20.031297 31696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "7918afab54d542da9394952f38842aa9"
format_stamp: "Formatted at 2026-09-27 03:20:20 on dist-test-slave-7j43"
server_key: "6e2e6548670bc4101a5bbe3b62193316"
server_key_iv: "dd42d6877b671c88728446f58cb1a36e"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.031371 31696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:20.045544 31696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:20.047207 31696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:20.047360 31696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:20.047566 31696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:20.047848 31696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:20.047904 31696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.047961 31696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:20.047986 31696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.053730 31696 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:33569
I20260927 03:20:20.053776 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:33569 every 8 connection(s)
I20260927 03:20:20.054059 31696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:20:20.057380 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31696
I20260927 03:20:20.057493 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:20:20.057667 17618 external_mini_cluster.cc:1428] Setting key 44044f624d21ee3a307194114833193c
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:20.062947 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.054715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:38735 (local address 127.17.52.190:33681)
0927 03:20:20.054777 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.054779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.054947 (+ 168us) server_negotiation.cc:407] Connection header received
0927 03:20:20.055390 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.055391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.055400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.055464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.056291 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.056848 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.057922 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.058170 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.058247 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.060367 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.060369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.060369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.060377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.061598 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.062004 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.062005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.062006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.062067 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.062321 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.062321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.062322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.062434 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.062505 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.062774 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.062858 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:20:20.063122 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.054682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:38735)
0927 03:20:20.054827 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.054866 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.055310 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.055530 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.055533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.056048 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:20.056217 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.056221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.056921 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.056921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.057837 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.057840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.058252 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.058254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.058393 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.058884 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:20.058900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:20.060272 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:20.061695 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.061701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.061703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.061927 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.062132 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.062133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.062134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.062240 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.062511 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:20.062516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:20.062644 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.062742 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.062901 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260927 03:20:20.063571 31825 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:20.063659 31825 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:20.063787 31825 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:20.064107 31366 ts_manager.cc:194] Registered new tserver with Master: 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
I20260927 03:20:20.064448 31366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:38735
I20260927 03:20:20.069245 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:20.075863 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.070586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.17.52.190:33681)
0927 03:20:20.070621 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.070623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.070768 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:20:20.070806 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.070806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.070815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.070868 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.071368 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.071827 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.072516 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.072761 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.072818 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.073922 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.073923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.073923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.073927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.075019 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.075309 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.075310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.075310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.075355 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.075468 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.075468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.075468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.075581 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.075648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.075733 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.075785 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:20:20.077574 31366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:20:20.079245 31366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:20:20.085335 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37637 (local address 127.0.0.1:54148)
0927 03:20:20.081509 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.081547 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.081624 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.081911 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.081912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.081913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.081973 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.081975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.082906 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.082907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.083981 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.083983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.085006 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.085007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.085082 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.085091 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.085182 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.085234 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":48,"threads_started":1}
I20260927 03:20:20.085383 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.17.52.130:37637)
0927 03:20:20.081495 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.081499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.081620 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:20:20.081735 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.081738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.081756 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.081816 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.082087 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.082784 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.084071 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.084844 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.084913 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.084976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.085215 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.085257 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260927 03:20:20.085569 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33569 (local address 127.0.0.1:44606)
0927 03:20:20.081712 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.081720 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.081750 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.081909 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.081911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.081913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.081973 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.081976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.083241 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.083241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.084173 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.084176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.085339 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.085340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.085414 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.085418 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.085461 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.085498 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":49,"threads_started":1}
I20260927 03:20:20.085820 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.17.52.131:33569)
0927 03:20:20.081732 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.081736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.081772 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:20:20.081815 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.081817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.081829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.081886 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.082055 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.083134 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.084274 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.085126 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.085280 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.085661 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.085690 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.085741 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260927 03:20:20.086270 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36979 (local address 127.0.0.1:41220)
0927 03:20:20.081583 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.081591 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.081631 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.082368 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.082369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.082371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.082404 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.082406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.083605 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.083606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.084746 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.084749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.085827 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.085829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.086111 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.086116 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.086171 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.086209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":60,"threads_started":1}
I20260927 03:20:20.086475 31759 tablet_service.cc:1467] Processing CreateTablet for tablet f5f68a29c10d41a0a80635c892c73961 (DEFAULT_TABLE table=test-table [id=283f6878021848d392684da978172688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.086469 31625 tablet_service.cc:1467] Processing CreateTablet for tablet f5f68a29c10d41a0a80635c892c73961 (DEFAULT_TABLE table=test-table [id=283f6878021848d392684da978172688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.086593 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.081373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.17.52.129:36979)
0927 03:20:20.082144 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.082148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.082157 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:20:20.082197 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.082199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.082214 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.082288 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.082476 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.083515 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.084939 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.085678 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.085745 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.085792 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.086449 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.086499 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":90,"threads_started":1}
I20260927 03:20:20.086817 31625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f68a29c10d41a0a80635c892c73961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.086812 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f68a29c10d41a0a80635c892c73961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.089289 31491 tablet_service.cc:1467] Processing CreateTablet for tablet f5f68a29c10d41a0a80635c892c73961 (DEFAULT_TABLE table=test-table [id=283f6878021848d392684da978172688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.089474 31839 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Bootstrap starting.
I20260927 03:20:20.089634 31491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f68a29c10d41a0a80635c892c73961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.089792 31840 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Bootstrap starting.
I20260927 03:20:20.090168 31839 tablet_bootstrap.cc:654] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.090399 31840 tablet_bootstrap.cc:654] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.090623 31839 log.cc:824] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.090950 31840 log.cc:824] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.091287 31839 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: No bootstrap required, opened a new log
I20260927 03:20:20.091349 31839 ts_tablet_manager.cc:1397] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:20.091707 31840 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: No bootstrap required, opened a new log
I20260927 03:20:20.091761 31840 ts_tablet_manager.cc:1397] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:20.092732 31839 raft_consensus.cc:348] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.092914 31839 raft_consensus.cc:374] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.092978 31839 raft_consensus.cc:729] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7918afab54d542da9394952f38842aa9, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.093232 31839 consensus_queue.cc:260] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.093516 31839 ts_tablet_manager.cc:1428] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:20.093568 31825 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
I20260927 03:20:20.093940 31840 raft_consensus.cc:348] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.094054 31840 raft_consensus.cc:374] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.094075 31840 raft_consensus.cc:729] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd9d5bf137a45cb9ad0a805ae0b8783, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.094192 31840 consensus_queue.cc:260] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.094472 31840 ts_tablet_manager.cc:1428] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:20:20.094503 31691 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
I20260927 03:20:20.094662 31843 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap starting.
I20260927 03:20:20.095342 31843 tablet_bootstrap.cc:654] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.095805 31843 log.cc:824] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.096433 31843 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: No bootstrap required, opened a new log
I20260927 03:20:20.096517 31843 ts_tablet_manager.cc:1397] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:20.097895 31843 raft_consensus.cc:348] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.098030 31843 raft_consensus.cc:374] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.098073 31843 raft_consensus.cc:729] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106fce8bcc4d4f9ca94ad9fac0a756eb, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.098218 31843 consensus_queue.cc:260] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.098453 31843 ts_tablet_manager.cc:1428] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:20.098484 31557 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
W20260927 03:20:20.157444 31692 tablet.cc:2369] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:20.178824 31844 raft_consensus.cc:482] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:20.178953 31844 raft_consensus.cc:504] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.179385 31844 leader_election.cc:288] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979)
I20260927 03:20:20.183185 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.179611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:51683 (local address 127.17.52.130:37637)
0927 03:20:20.179686 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.179688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.179699 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:20:20.179729 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.179730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.179743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.179809 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.180007 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.180795 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182256 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.182841 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182914 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.182922 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.183077 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.183114 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:20:20.183185 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.179685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:53681 (local address 127.17.52.129:36979)
0927 03:20:20.179732 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.179734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.179858 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:20:20.179894 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.179895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.179902 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.179945 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.180100 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.180812 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182257 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.182836 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.182922 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.183077 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.183115 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:20:20.183194 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.179541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37637 (local address 127.17.52.131:51683)
0927 03:20:20.179596 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.179608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.179651 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.179833 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.179834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.179836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.179909 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.179911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.180893 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.180893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.182166 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.182169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.182919 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.182998 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.182999 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.183056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.183101 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":33}
I20260927 03:20:20.183204 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.179554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36979 (local address 127.17.52.131:53681)
0927 03:20:20.179747 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.179756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.179837 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.179966 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.179967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.179968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.180028 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.180029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.180893 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.180893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.182166 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.182169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.182919 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.182920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.182995 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.182999 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.183056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.183101 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":79,"thread_start_us":111,"threads_started":1}
I20260927 03:20:20.183562 31645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f68a29c10d41a0a80635c892c73961" candidate_uuid: "7918afab54d542da9394952f38842aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" is_pre_election: true
I20260927 03:20:20.183736 31645 raft_consensus.cc:2454] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7918afab54d542da9394952f38842aa9 in term 0.
I20260927 03:20:20.183735 31511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f68a29c10d41a0a80635c892c73961" candidate_uuid: "7918afab54d542da9394952f38842aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" is_pre_election: true
I20260927 03:20:20.183920 31511 raft_consensus.cc:2454] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7918afab54d542da9394952f38842aa9 in term 0.
I20260927 03:20:20.183974 31713 leader_election.cc:302] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9; no voters:
I20260927 03:20:20.184202 31844 raft_consensus.cc:2781] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:20.184249 31844 raft_consensus.cc:482] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:20.184265 31844 raft_consensus.cc:3037] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.184969 31844 raft_consensus.cc:504] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.185106 31844 leader_election.cc:288] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979)
I20260927 03:20:20.185266 31645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f68a29c10d41a0a80635c892c73961" candidate_uuid: "7918afab54d542da9394952f38842aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
I20260927 03:20:20.185271 31511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f68a29c10d41a0a80635c892c73961" candidate_uuid: "7918afab54d542da9394952f38842aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
I20260927 03:20:20.185341 31645 raft_consensus.cc:3037] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.185341 31511 raft_consensus.cc:3037] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.186218 31511 raft_consensus.cc:2454] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7918afab54d542da9394952f38842aa9 in term 1.
I20260927 03:20:20.186218 31645 raft_consensus.cc:2454] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7918afab54d542da9394952f38842aa9 in term 1.
I20260927 03:20:20.186460 31715 leader_election.cc:302] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb, 7918afab54d542da9394952f38842aa9; no voters:
I20260927 03:20:20.186569 31844 raft_consensus.cc:2781] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:20.186724 31844 raft_consensus.cc:686] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 7918afab54d542da9394952f38842aa9, State: Running, Role: LEADER
I20260927 03:20:20.186836 31844 consensus_queue.cc:237] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.187659 31366 catalog_manager.cc:5495] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7918afab54d542da9394952f38842aa9 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "7918afab54d542da9394952f38842aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } health_report { overall_health: HEALTHY } } }
I20260927 03:20:20.200686 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.198518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.17.52.131:33569)
0927 03:20:20.198573 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.198575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.198582 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:20:20.198631 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.198632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.198646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.198706 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:20.198818 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.199268 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.200011 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.200183 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.200233 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.200351 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:20.200437 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:20.200492 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.200518 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.200534 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:20:20.202322 31645 raft_consensus.cc:1261] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Refusing update from remote peer 7918afab54d542da9394952f38842aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:20.202322 31511 raft_consensus.cc:1261] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Refusing update from remote peer 7918afab54d542da9394952f38842aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:20.202641 31844 consensus_queue.cc:1035] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:20.202800 31849 consensus_queue.cc:1035] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:20.205744 31854 mvcc.cc:205] Tried to move back new op lower bound from 7333802885946281984 to 7333802885885968384. Current Snapshot: MvccSnapshot[applied={T|T < 7333802885946281984}]
I20260927 03:20:20.207177 31364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:20.207274 31364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:20.208021 31364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Sending DeleteTablet for 3 replicas of tablet f5f68a29c10d41a0a80635c892c73961
I20260927 03:20:20.208318 31625 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f68a29c10d41a0a80635c892c73961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:54148
I20260927 03:20:20.208343 31491 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f68a29c10d41a0a80635c892c73961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:41220
I20260927 03:20:20.208343 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f68a29c10d41a0a80635c892c73961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44606
I20260927 03:20:20.208527 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31428
I20260927 03:20:20.208657 31864 tablet_replica.cc:330] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: stopping tablet replica
I20260927 03:20:20.208771 31864 raft_consensus.cc:2229] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:20:20.208909 31864 raft_consensus.cc:2258] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.209283 31864 ts_tablet_manager.cc:1905] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:20.210278 31863 tablet_replica.cc:330] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: stopping tablet replica
I20260927 03:20:20.210388 31863 raft_consensus.cc:2229] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:20.210486 31863 raft_consensus.cc:2258] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.210661 31864 ts_tablet_manager.cc:1918] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:20.210752 31864 log.cc:1196] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/f5f68a29c10d41a0a80635c892c73961
I20260927 03:20:20.211009 31864 ts_tablet_manager.cc:1939] T f5f68a29c10d41a0a80635c892c73961 P 7918afab54d542da9394952f38842aa9: Deleting consensus metadata
I20260927 03:20:20.211380 31351 catalog_manager.cc:4841] TS 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): tablet f5f68a29c10d41a0a80635c892c73961 (table test-table [id=283f6878021848d392684da978172688]) successfully deleted
I20260927 03:20:20.212878 31863 ts_tablet_manager.cc:1905] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:20.214025 31863 ts_tablet_manager.cc:1918] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:20.214110 31863 log.cc:1196] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/f5f68a29c10d41a0a80635c892c73961
I20260927 03:20:20.214375 31863 ts_tablet_manager.cc:1939] T f5f68a29c10d41a0a80635c892c73961 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting consensus metadata
W20260927 03:20:20.214740 31353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:36979 (error 108)
I20260927 03:20:20.214764 31351 catalog_manager.cc:4841] TS 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): tablet f5f68a29c10d41a0a80635c892c73961 (table test-table [id=283f6878021848d392684da978172688]) successfully deleted
I20260927 03:20:20.215036 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31562
I20260927 03:20:20.215583 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.215081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36979 (local address 127.0.0.1:41224)
0927 03:20:20.215515 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.215557 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260927 03:20:20.215786 31353 catalog_manager.cc:4584] TS 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5f68a29c10d41a0a80635c892c73961: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36979: connect: Connection refused (error 111)
I20260927 03:20:20.221844 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31696
I20260927 03:20:20.226903 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31334
I20260927 03:20:20.231781 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--webserver_interface=127.17.52.190
--webserver_port=42739
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:20.312213 31866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:20.312422 31866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:20.312444 31866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:20.313593 31866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:20.313647 31866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:20.313663 31866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:20.313676 31866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:20.313688 31866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:20.315081 31866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=42739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:20.315272 31866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:20.315433 31866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:20.325125 31872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.325153 31875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.325125 31873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.325362 31866 server_base.cc:1034] running on GCE node
I20260927 03:20:20.325614 31866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:20.325980 31866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:20.327142 31866 hybrid_clock.cc:648] HybridClock initialized: now 1790479220327121 us; error 45 us; skew 500 ppm
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:20.333601 31866 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:20.334095 31866 webserver.cc:460] Webserver started at http://127.17.52.190:42739/ using document root <none> and password file <none>
I20260927 03:20:20.334374 31866 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:20.334462 31866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:20.335870 31866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:20:20.336668 31882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.336848 31866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:20.336952 31866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "6206ce6730234cdfa9b82d8a5b72e890"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "3bac34e8e0198d13cd143eb7bd9fcc3e"
server_key_iv: "46101d0edbe9cee6b037ad2bc14ce683"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.337378 31866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:20.366200 31866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:20.368155 31866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:20.368299 31866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:20.372279 31866 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33681
I20260927 03:20:20.372363 31934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33681 every 8 connection(s)
I20260927 03:20:20.372789 31866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:20.374392 31935 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:20:20.374974 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap starting.
I20260927 03:20:20.376760 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31866
I20260927 03:20:20.377022 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:36979
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=43237
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:20.383492 31935 log.cc:824] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.385512 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:20.385816 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap complete.
I20260927 03:20:20.388077 31935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:20.388326 31935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.388527 31935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:20.388638 31935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:20.388695 31935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:20.388742 31935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:20:20.389753 31935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:20.389874 31935 leader_election.cc:302] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6206ce6730234cdfa9b82d8a5b72e890; no voters:
I20260927 03:20:20.390053 31935 leader_election.cc:288] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:20:20.390134 31939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:20:20.390283 31939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 LEADER]: Becoming Leader. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Running, Role: LEADER
I20260927 03:20:20.390411 31935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:20.390441 31939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:20.390789 31940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:20.390866 31940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:20.391222 31945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:20.391253 31941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6206ce6730234cdfa9b82d8a5b72e890. Latest consensus state: current_term: 2 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:20.391433 31941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:20.392320 31945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:20.392468 31945 catalog_manager.cc:1259] Loaded cluster ID: a34a36b174584577b7176d3e0675bab0
I20260927 03:20:20.392726 31945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:20:20.393029 31945 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 03:20:20.397699 31945 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 03:20:20.400074 31945 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:20.400321 31945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Loaded TSK: 0
I20260927 03:20:20.400578 31945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:20:20.459062 31937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:20.459247 31937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:20.459270 31937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:20.460389 31937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:20.460427 31937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:20.460453 31937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:20.461875 31937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:36979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=43237
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:20.462095 31937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:20.462338 31937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:20.467168 31963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.467283 31937 server_base.cc:1034] running on GCE node
W20260927 03:20:20.467152 31962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.467140 31965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.467713 31937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:20.468019 31937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:20.469169 31937 hybrid_clock.cc:648] HybridClock initialized: now 1790479220469150 us; error 29 us; skew 500 ppm
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:20.472216 31937 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:20.472743 31937 webserver.cc:460] Webserver started at http://127.17.52.129:43237/ using document root <none> and password file <none>
I20260927 03:20:20.473016 31937 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:20.473102 31937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:20.474741 31937 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:20:20.475487 31972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.475683 31937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:20.475778 31937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "9aacd72f389f243ae2a2df239fc3db90"
server_key_iv: "1e65ac64bf6e092468b96a3d99c636bc"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.476172 31937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:20.504676 31937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:20.507066 31937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:20.507216 31937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:20.507462 31937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:20.507956 31979 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:20:20.509445 31937 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:20:20.509508 31937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:20:20.509543 31937 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:20:20.510387 31937 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:20:20.510437 31937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:20.510504 31979 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap starting.
I20260927 03:20:20.517235 31937 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:36979
I20260927 03:20:20.517395 32086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:36979 every 8 connection(s)
I20260927 03:20:20.517676 31937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:20.521137 31979 log.cc:824] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.522498 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 31937
I20260927 03:20:20.522779 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:37637
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=41425
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:20.523777 31979 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:20.524078 31979 tablet_bootstrap.cc:492] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap complete.
I20260927 03:20:20.525673 31979 ts_tablet_manager.cc:1397] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:20.526937 31979 raft_consensus.cc:348] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.527220 31979 raft_consensus.cc:729] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106fce8bcc4d4f9ca94ad9fac0a756eb, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.531742 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.518466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59575 (local address 127.17.52.190:33681)
0927 03:20:20.518659 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.518669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.518909 (+ 240us) server_negotiation.cc:407] Connection header received
0927 03:20:20.519720 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.519728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.519787 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.519936 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.520964 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.522379 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.523939 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.524559 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.525156 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.527510 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.527514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.527514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.527527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.529746 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.530217 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.530221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.530221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.530358 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.530624 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.530625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.530625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.530938 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.531037 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.531353 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.531509 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260927 03:20:20.531621 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.518358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:59575)
0927 03:20:20.518716 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.518768 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.519484 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.520003 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.520008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.520505 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:20.520835 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.520842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.522512 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.522513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.523814 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.523818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.524717 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.524719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.524860 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.525464 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:20.525473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:20.527382 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:20.529844 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.529850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.529852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.530129 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.530434 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.530435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.530436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.530551 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.531013 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:20.531018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:20.531158 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.531265 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.531420 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":79,"threads_started":1}
I20260927 03:20:20.532766 31979 consensus_queue.cc:260] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } }
I20260927 03:20:20.533223 31979 ts_tablet_manager.cc:1428] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260927 03:20:20.533596 32087 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:20.533684 32087 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:20.533861 32087 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:20.534686 31899 ts_manager.cc:194] Registered new tserver with Master: 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979)
I20260927 03:20:20.535933 31899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:59575
I20260927 03:20:20.536923 32087 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:20:20.543658 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.535548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.17.52.129:36979)
0927 03:20:20.535698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.535703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.536379 (+ 676us) server_negotiation.cc:407] Connection header received
0927 03:20:20.536414 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.536416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.536433 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.536510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.536971 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.537769 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.538511 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.538802 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.538887 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.540930 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.540937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.540938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.540944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.542415 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.542784 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.542787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.542788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.542851 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.543062 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.543063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.543063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.543198 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.543308 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.543486 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.543570 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260927 03:20:20.543658 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.535321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36979 (local address 127.0.0.1:41230)
0927 03:20:20.536160 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.536221 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.536287 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.536512 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.536518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.536836 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:20.536878 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.536880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.537862 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.537863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.538421 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.538423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.538894 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.538896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.538976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.539428 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:20.539432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:20.540841 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:20.542500 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.542507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.542508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.542702 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.542920 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.542921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.542921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.542986 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.543293 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:20.543299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:20.543350 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.543451 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.543565 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":81,"threads_started":1}
I20260927 03:20:20.544042 32021 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f68a29c10d41a0a80635c892c73961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:41230
I20260927 03:20:20.544605 32096 tablet_replica.cc:330] stopping tablet replica
I20260927 03:20:20.544704 32096 raft_consensus.cc:2229] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:20.544771 32096 raft_consensus.cc:2258] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.545034 32096 ts_tablet_manager.cc:1905] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:20.546773 32096 ts_tablet_manager.cc:1918] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:20.546860 32096 log.cc:1196] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/f5f68a29c10d41a0a80635c892c73961
I20260927 03:20:20.547078 32096 ts_tablet_manager.cc:1939] T f5f68a29c10d41a0a80635c892c73961 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Deleting consensus metadata
I20260927 03:20:20.547545 31886 catalog_manager.cc:4841] TS 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979): tablet f5f68a29c10d41a0a80635c892c73961 (table test-table [id=283f6878021848d392684da978172688]) successfully deleted
W20260927 03:20:20.610446 32092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:20.610639 32092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:20.610662 32092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:20.611729 32092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:20.611763 32092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:20.611788 32092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:20.613230 32092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=41425
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:20.613422 32092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:20.613610 32092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:20.618527 32102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.618628 32092 server_base.cc:1034] running on GCE node
W20260927 03:20:20.618563 32104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.618728 32101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.618924 32092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:20.619159 32092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:20.620338 32092 hybrid_clock.cc:648] HybridClock initialized: now 1790479220620321 us; error 33 us; skew 500 ppm
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:20.622905 32092 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:20.623373 32092 webserver.cc:460] Webserver started at http://127.17.52.130:41425/ using document root <none> and password file <none>
I20260927 03:20:20.623613 32092 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:20.623697 32092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:20.625089 32092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:20:20.625659 32111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.625842 32092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.625941 32092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "308c50f6f74d313ccd926334f3918c6d"
server_key_iv: "c76ac045823b6465793cf4408d00033d"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.626276 32092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:20.637259 32092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:20.639473 32092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:20.639626 32092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:20.639868 32092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:20.640188 32092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:20.640234 32092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.640269 32092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:20.640300 32092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.645979 32092 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:37637
I20260927 03:20:20.646026 32224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:37637 every 8 connection(s)
I20260927 03:20:20.646346 32092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:20:20.650610 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32092
I20260927 03:20:20.650915 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:33569
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=42377
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:20.655835 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.647008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36531 (local address 127.17.52.190:33681)
0927 03:20:20.647061 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.647064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.647258 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:20:20.647712 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.647713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.647725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.647774 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.648712 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.649207 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.650150 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.650342 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.650391 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.652933 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.652934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.652935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.652943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.654309 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.654832 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.654833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.654834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.654905 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.655199 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.655200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.655200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.655311 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.655360 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.655699 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.655767 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:20:20.656142 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.646997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:36531)
0927 03:20:20.647145 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.647178 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.647635 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.647845 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.647848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.648386 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:20.648628 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.648635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.649283 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.649283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.650081 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.650083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.650444 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.650445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.650599 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.651170 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:20.651180 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:20.652845 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:20.654393 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.654400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.654402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.654729 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.654976 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.654977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.654977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.655128 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.655376 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:20.655383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:20.655554 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.655682 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.655867 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260927 03:20:20.656672 32225 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:20.656780 32225 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:20.656977 32225 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:20.657383 31899 ts_manager.cc:194] Registered new tserver with Master: 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637)
I20260927 03:20:20.657755 31899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:36531
W20260927 03:20:20.732847 32228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:20.733038 32228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:20.733062 32228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:20.734184 32228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:20.734225 32228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:20.734277 32228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:20.735700 32228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:33569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=42377
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:20.735942 32228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:20.736164 32228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:20.740918 32236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.740895 32234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:20.740918 32233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:20.741256 32228 server_base.cc:1034] running on GCE node
I20260927 03:20:20.741463 32228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:20.741707 32228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:20.742857 32228 hybrid_clock.cc:648] HybridClock initialized: now 1790479220742844 us; error 22 us; skew 500 ppm
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:20.745261 32228 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:20.745656 32228 webserver.cc:460] Webserver started at http://127.17.52.131:42377/ using document root <none> and password file <none>
I20260927 03:20:20.745875 32228 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:20.745982 32228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:20.747264 32228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:20.747783 32243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.747928 32228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.748028 32228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "7918afab54d542da9394952f38842aa9"
format_stamp: "Formatted at 2026-09-27 03:20:20 on dist-test-slave-7j43"
server_key: "6e2e6548670bc4101a5bbe3b62193316"
server_key_iv: "dd42d6877b671c88728446f58cb1a36e"
server_key_version: "encryptionkey@0"
I20260927 03:20:20.748338 32228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:20.755336 32228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:20.757478 32228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:20.757632 32228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:20.757865 32228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:20.758150 32228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:20.758203 32228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.758244 32228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:20.758272 32228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:20.763729 32228 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:33569
I20260927 03:20:20.763845 32356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:33569 every 8 connection(s)
I20260927 03:20:20.764140 32228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:20:20.766335 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32228
Sep 27 03:20:20 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479220, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:20.773473 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.766683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.17.52.190:33681)
0927 03:20:20.766793 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.766795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.766807 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:20:20.766889 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.766889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.766899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.766946 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.767586 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.768325 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.769285 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.769558 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.769628 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.770779 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.770780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.770780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.770785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.772146 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.772487 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.772488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.772488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.772571 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.772821 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.772822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.772822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.772952 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.773133 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.773206 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.773294 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260927 03:20:20.773973 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.764898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:52485 (local address 127.17.52.190:33681)
0927 03:20:20.764980 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.764982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.765150 (+ 168us) server_negotiation.cc:407] Connection header received
0927 03:20:20.765581 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.765582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.765592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.765645 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:20.766516 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.767281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.768660 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.768955 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.769027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.771086 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:20.771087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:20.771087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:20.771094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:20.772511 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.772992 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.772993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.772993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.773033 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:20.773323 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:20.773324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:20.773324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:20.773462 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:20.773535 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.773851 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.773919 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:20:20.774140 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.764861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:52485)
0927 03:20:20.765011 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.765050 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.765502 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.765695 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.765698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.766221 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:20.766431 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.766437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.767388 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.767389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.768548 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.768551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.769037 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.769038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.769134 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.769595 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:20.769605 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:20.771019 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:20.772603 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.772610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.772612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.772921 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.773093 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:20.773094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:20.773094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:20.773219 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:20.773542 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:20.773549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:20.773703 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.773824 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.773951 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260927 03:20:20.774525 32357 heartbeater.cc:347] Connected to a master server at 127.17.52.190:33681
I20260927 03:20:20.774632 32357 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:20.774812 32357 heartbeater.cc:511] Master 127.17.52.190:33681 requested a full tablet report, sending...
I20260927 03:20:20.775161 31898 ts_manager.cc:194] Registered new tserver with Master: 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
I20260927 03:20:20.775530 31898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:52485
I20260927 03:20:20.785561 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:20.788151 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.785865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.17.52.190:33681)
0927 03:20:20.785916 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.785918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.786042 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:20:20.786088 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.786088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.786118 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.786177 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:20.786278 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.786715 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.787540 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.787706 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.787752 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.787917 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:20.788002 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:20.788058 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.788091 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.788104 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:20:20.788407 31898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:20:20.790016 31898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:20:20.792163 32021 tablet_service.cc:1467] Processing CreateTablet for tablet 18682963dd434b03ac44fbc8faebc538 (DEFAULT_TABLE table=test-table [id=26b51bab0e6a4ed69c57e032af5d18c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.792363 32021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18682963dd434b03ac44fbc8faebc538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.793694 31979 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap starting.
I20260927 03:20:20.794011 31979 tablet_bootstrap.cc:654] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.794588 31979 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: No bootstrap required, opened a new log
I20260927 03:20:20.794636 31979 ts_tablet_manager.cc:1397] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:20.794986 31979 raft_consensus.cc:348] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.795061 31979 raft_consensus.cc:374] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.795101 31979 raft_consensus.cc:729] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106fce8bcc4d4f9ca94ad9fac0a756eb, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.795146 31979 consensus_queue.cc:260] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.795219 31979 ts_tablet_manager.cc:1428] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:20.796029 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.791912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37637 (local address 127.0.0.1:54152)
0927 03:20:20.791945 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.791955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.791992 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.792536 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.792538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.792540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.792603 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.792605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.793911 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.793912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.794957 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.794959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.795845 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.795845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.795899 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.795913 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.795952 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.795983 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:20:20.796095 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.792225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.17.52.130:37637)
0927 03:20:20.792373 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.792378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.792388 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:20:20.792427 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.792430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.792447 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.792529 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.792671 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.793796 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.795039 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.795773 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.795844 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.795899 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.795975 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.796012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 03:20:20.796450 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.792014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33569 (local address 127.0.0.1:44628)
0927 03:20:20.792195 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.792205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.792251 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.792896 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.792897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.792899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.792939 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.792940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.794187 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.794189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.795290 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.795293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.796230 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.796231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.796294 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.796297 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.796349 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.796387 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":135,"threads_started":1}
I20260927 03:20:20.796507 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.792410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.17.52.131:33569)
0927 03:20:20.792725 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.792731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.792740 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:20:20.792776 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.792778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.792792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.793007 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.793068 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.794071 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.795385 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.796061 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.796137 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.796193 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.796391 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.796432 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260927 03:20:20.796799 32159 tablet_service.cc:1467] Processing CreateTablet for tablet 18682963dd434b03ac44fbc8faebc538 (DEFAULT_TABLE table=test-table [id=26b51bab0e6a4ed69c57e032af5d18c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.797133 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18682963dd434b03ac44fbc8faebc538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.797070 32291 tablet_service.cc:1467] Processing CreateTablet for tablet 18682963dd434b03ac44fbc8faebc538 (DEFAULT_TABLE table=test-table [id=26b51bab0e6a4ed69c57e032af5d18c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:20.797353 32291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18682963dd434b03ac44fbc8faebc538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:20.800568 32367 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Bootstrap starting.
I20260927 03:20:20.801326 32367 tablet_bootstrap.cc:654] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.801787 32367 log.cc:824] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.802093 32368 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Bootstrap starting.
I20260927 03:20:20.802469 32367 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: No bootstrap required, opened a new log
I20260927 03:20:20.802522 32367 ts_tablet_manager.cc:1397] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:20.802692 32368 tablet_bootstrap.cc:654] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:20.803138 32368 log.cc:824] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:20.803887 32368 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: No bootstrap required, opened a new log
I20260927 03:20:20.803970 32368 ts_tablet_manager.cc:1397] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:20.804054 32367 raft_consensus.cc:348] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.804188 32367 raft_consensus.cc:374] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.804209 32367 raft_consensus.cc:729] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7918afab54d542da9394952f38842aa9, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.804344 32367 consensus_queue.cc:260] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.804647 32357 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
I20260927 03:20:20.804657 32367 ts_tablet_manager.cc:1428] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:20.805627 32368 raft_consensus.cc:348] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.805768 32368 raft_consensus.cc:374] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:20.805814 32368 raft_consensus.cc:729] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd9d5bf137a45cb9ad0a805ae0b8783, State: Initialized, Role: FOLLOWER
I20260927 03:20:20.805920 32368 consensus_queue.cc:260] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.806196 32368 ts_tablet_manager.cc:1428] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:20.806219 32225 heartbeater.cc:503] Master 127.17.52.190:33681 was elected leader, sending a full tablet report...
W20260927 03:20:20.896872 32226 tablet.cc:2369] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:20.910267 32093 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:20.910423 32093 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.910751 32093 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
I20260927 03:20:20.913643 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.911084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:47587 (local address 127.17.52.130:37637)
0927 03:20:20.911134 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.911137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.911149 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:20:20.911179 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.911180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.911190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.911256 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.911429 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.911904 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.912909 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.913274 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.913367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.913376 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.913537 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.913575 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:20:20.913643 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.910964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37637 (local address 127.17.52.129:47587)
0927 03:20:20.911023 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.911036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.911099 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.911273 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.911274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.911276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.911350 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.911352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.911984 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.911985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.912824 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.912826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.913363 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.913364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.913443 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.913463 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.913531 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.913571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:20:20.913980 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.911081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33569 (local address 127.17.52.129:40543)
0927 03:20:20.911255 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.911266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:20.911330 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:20.911469 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:20.911470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:20.911471 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:20.911562 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.911563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.912183 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.912184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.912993 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:20.912995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.913565 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.913566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:20.913646 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.913649 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:20.913859 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 03:20:20.913897 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260927 03:20:20.914004 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.911226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:40543 (local address 127.17.52.131:33569)
0927 03:20:20.911273 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.911275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.911316 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:20:20.911406 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.911408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.911418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.911467 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:20.911635 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.912101 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.913079 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.913469 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.913552 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.913563 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.913891 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.913943 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:20:20.913996 32179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18682963dd434b03ac44fbc8faebc538" candidate_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" is_pre_election: true
I20260927 03:20:20.914168 32179 raft_consensus.cc:2454] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106fce8bcc4d4f9ca94ad9fac0a756eb in term 0.
I20260927 03:20:20.914361 32311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18682963dd434b03ac44fbc8faebc538" candidate_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7918afab54d542da9394952f38842aa9" is_pre_election: true
I20260927 03:20:20.914415 31974 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb, 6fd9d5bf137a45cb9ad0a805ae0b8783; no voters:
I20260927 03:20:20.914494 32311 raft_consensus.cc:2454] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106fce8bcc4d4f9ca94ad9fac0a756eb in term 0.
I20260927 03:20:20.914572 32093 raft_consensus.cc:2781] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:20.914633 32093 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:20.914662 32093 raft_consensus.cc:3037] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.915351 32093 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.915477 32093 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 1 election: Requested vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
I20260927 03:20:20.915655 32179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18682963dd434b03ac44fbc8faebc538" candidate_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
I20260927 03:20:20.915647 32311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18682963dd434b03ac44fbc8faebc538" candidate_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7918afab54d542da9394952f38842aa9"
I20260927 03:20:20.915719 32311 raft_consensus.cc:3037] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.915743 32179 raft_consensus.cc:3037] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:20.916417 32179 raft_consensus.cc:2454] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106fce8bcc4d4f9ca94ad9fac0a756eb in term 1.
I20260927 03:20:20.916424 32311 raft_consensus.cc:2454] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106fce8bcc4d4f9ca94ad9fac0a756eb in term 1.
I20260927 03:20:20.916658 31974 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb, 7918afab54d542da9394952f38842aa9; no voters:
I20260927 03:20:20.916790 32093 raft_consensus.cc:2781] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:20.916976 32093 raft_consensus.cc:686] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 LEADER]: Becoming Leader. State: Replica: 106fce8bcc4d4f9ca94ad9fac0a756eb, State: Running, Role: LEADER
I20260927 03:20:20.917042 32093 consensus_queue.cc:237] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:20.917833 31899 catalog_manager.cc:5495] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129). New cstate: current_term: 1 leader_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } health_report { overall_health: HEALTHY } } }
I20260927 03:20:20.940997 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.938784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.17.52.129:36979)
0927 03:20:20.938843 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:20.938844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:20.938852 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:20:20.938899 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:20.938900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:20.938916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:20.938965 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:20.939055 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.939533 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.940270 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:20.940481 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:20.940532 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:20.940671 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:20.940762 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:20.940820 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:20.940871 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:20:20.940896 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:20:20.942400 32179 raft_consensus.cc:1261] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Refusing update from remote peer 106fce8bcc4d4f9ca94ad9fac0a756eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:20.942399 32311 raft_consensus.cc:1261] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Refusing update from remote peer 106fce8bcc4d4f9ca94ad9fac0a756eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:20.942706 32093 consensus_queue.cc:1035] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:20.942866 32374 consensus_queue.cc:1035] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:20.945709 32379 mvcc.cc:205] Tried to move back new op lower bound from 7333802888977694720 to 7333802888876732416. Current Snapshot: MvccSnapshot[applied={T|T < 7333802888977694720}]
I20260927 03:20:20.948061 31899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:20.948184 31899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:20.948971 31899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Sending DeleteTablet for 3 replicas of tablet 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:20.949270 32021 tablet_service.cc:1514] Processing DeleteTablet for tablet 18682963dd434b03ac44fbc8faebc538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:41230
I20260927 03:20:20.949294 32159 tablet_service.cc:1514] Processing DeleteTablet for tablet 18682963dd434b03ac44fbc8faebc538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:54152
I20260927 03:20:20.949325 32291 tablet_service.cc:1514] Processing DeleteTablet for tablet 18682963dd434b03ac44fbc8faebc538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:20 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:44628
I20260927 03:20:20.949404 32096 tablet_replica.cc:330] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: stopping tablet replica
I20260927 03:20:20.949460 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31937
I20260927 03:20:20.949473 32096 raft_consensus.cc:2229] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:20:20.949559 32096 raft_consensus.cc:2258] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.950474 32387 tablet_replica.cc:330] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: stopping tablet replica
I20260927 03:20:20.950574 32387 raft_consensus.cc:2229] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:20.950666 32387 raft_consensus.cc:2258] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.950837 32386 tablet_replica.cc:330] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: stopping tablet replica
I20260927 03:20:20.950923 32386 raft_consensus.cc:2229] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:20.951013 32386 raft_consensus.cc:2258] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:20.951277 32387 ts_tablet_manager.cc:1905] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:20.951437 32386 ts_tablet_manager.cc:1905] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:20.952926 32387 ts_tablet_manager.cc:1918] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:20.952996 32386 ts_tablet_manager.cc:1918] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:20.953011 32387 log.cc:1196] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/18682963dd434b03ac44fbc8faebc538
I20260927 03:20:20.953063 32386 log.cc:1196] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/18682963dd434b03ac44fbc8faebc538
I20260927 03:20:20.953311 32387 ts_tablet_manager.cc:1939] T 18682963dd434b03ac44fbc8faebc538 P 7918afab54d542da9394952f38842aa9: Deleting consensus metadata
I20260927 03:20:20.953351 32386 ts_tablet_manager.cc:1939] T 18682963dd434b03ac44fbc8faebc538 P 6fd9d5bf137a45cb9ad0a805ae0b8783: Deleting consensus metadata
I20260927 03:20:20.953773 31884 catalog_manager.cc:4841] TS 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): tablet 18682963dd434b03ac44fbc8faebc538 (table test-table [id=26b51bab0e6a4ed69c57e032af5d18c7]) successfully deleted
I20260927 03:20:20.953850 31884 catalog_manager.cc:4841] TS 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): tablet 18682963dd434b03ac44fbc8faebc538 (table test-table [id=26b51bab0e6a4ed69c57e032af5d18c7]) successfully deleted
W20260927 03:20:20.955824 31886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:36979 (error 108)
I20260927 03:20:20.956202 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 32092
I20260927 03:20:20.956256 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:20.956150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:36979 (local address 127.0.0.1:41240)
0927 03:20:20.956187 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:20.956220 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:20:20.956838 31886 catalog_manager.cc:4584] TS 106fce8bcc4d4f9ca94ad9fac0a756eb (127.17.52.129:36979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18682963dd434b03ac44fbc8faebc538: Network error: Client connection negotiation failed: client connection to 127.17.52.129:36979: connect: Connection refused (error 111)
I20260927 03:20:20.961112 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 32228
I20260927 03:20:20.965370 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 31866
I20260927 03:20:20.971026 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--webserver_interface=127.17.52.190
--webserver_port=42739
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:21.050953 32389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:21.051178 32389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:21.051213 32389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:21.052320 32389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:21.052381 32389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:21.052405 32389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:21.052426 32389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:21.052456 32389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:21.054100 32389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=42739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:21.054380 32389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:21.054600 32389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:21.059393 32397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.059392 32395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.059399 32394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:21.059614 32389 server_base.cc:1034] running on GCE node
I20260927 03:20:21.059855 32389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:21.060150 32389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:21.061306 32389 hybrid_clock.cc:648] HybridClock initialized: now 1790479221061298 us; error 43 us; skew 500 ppm
Sep 27 03:20:21 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:21.063683 32389 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:21.064132 32389 webserver.cc:460] Webserver started at http://127.17.52.190:42739/ using document root <none> and password file <none>
I20260927 03:20:21.064347 32389 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:21.064414 32389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:21.065738 32389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:21.066380 32404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.066545 32389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:20:21.066639 32389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "6206ce6730234cdfa9b82d8a5b72e890"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "3bac34e8e0198d13cd143eb7bd9fcc3e"
server_key_iv: "46101d0edbe9cee6b037ad2bc14ce683"
server_key_version: "encryptionkey@0"
I20260927 03:20:21.066987 32389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:21.078418 32389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:21.080662 32389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:21.080816 32389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:21.084692 32389 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33681
I20260927 03:20:21.084736 32456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33681 every 8 connection(s)
I20260927 03:20:21.085140 32389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:21.085531 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32389
I20260927 03:20:21.085814 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:36979
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=43237
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:21.086844 32457 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:20:21.087510 32457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap starting.
I20260927 03:20:21.098869 32457 log.cc:824] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:21.101714 32457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:21.102053 32457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890: Bootstrap complete.
I20260927 03:20:21.105345 32457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:21.105731 32457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Initialized, Role: FOLLOWER
I20260927 03:20:21.105926 32457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:21.106003 32457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:21.106043 32457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:21.106089 32457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:20:21.107398 32457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:21.107504 32457 leader_election.cc:302] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6206ce6730234cdfa9b82d8a5b72e890; no voters:
I20260927 03:20:21.107661 32457 leader_election.cc:288] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:20:21.107735 32461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:20:21.107887 32461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [term 3 LEADER]: Becoming Leader. State: Replica: 6206ce6730234cdfa9b82d8a5b72e890, State: Running, Role: LEADER
I20260927 03:20:21.107910 32457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:21.107990 32461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } }
I20260927 03:20:21.108338 32462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:21.108413 32462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:21.108637 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6206ce6730234cdfa9b82d8a5b72e890. Latest consensus state: current_term: 3 leader_uuid: "6206ce6730234cdfa9b82d8a5b72e890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6206ce6730234cdfa9b82d8a5b72e890" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33681 } } }
I20260927 03:20:21.108764 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206ce6730234cdfa9b82d8a5b72e890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:21.108903 32469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:21.109730 32469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:21.109840 32469 catalog_manager.cc:1259] Loaded cluster ID: a34a36b174584577b7176d3e0675bab0
I20260927 03:20:21.109872 32469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 03:20:21.110355 32469 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff171daf9ad google::LogMessage::Fail() at ??:0
@ 0x7ff171db288a google::LogMessage::SendToLog() at ??:0
@ 0x7ff171daf4de google::LogMessage::Flush() at ??:0
@ 0x7ff171db0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff179cd00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff179cd98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff173c7d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff173c72954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff174ad66db start_thread at ??:0
@ 0x7ff1708ad61f clone at ??:0
W20260927 03:20:21.209327 32459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:21.209612 32459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:21.209650 32459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:21.211355 32459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:21.211416 32459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:21.211455 32459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:21.213600 32459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:36979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=43237
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:21.213852 32459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:21.214105 32459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:21.219422 32494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.219432 32493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.219455 32496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:21.219730 32459 server_base.cc:1034] running on GCE node
I20260927 03:20:21.219945 32459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:21.220206 32459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:21.221354 32459 hybrid_clock.cc:648] HybridClock initialized: now 1790479221221330 us; error 41 us; skew 500 ppm
Sep 27 03:20:21 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:21.223789 32459 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:21.224208 32459 webserver.cc:460] Webserver started at http://127.17.52.129:43237/ using document root <none> and password file <none>
I20260927 03:20:21.224448 32459 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:21.224519 32459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:21.225957 32459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:21.226696 32503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.226887 32459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:21.226994 32459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "9aacd72f389f243ae2a2df239fc3db90"
server_key_iv: "1e65ac64bf6e092468b96a3d99c636bc"
server_key_version: "encryptionkey@0"
I20260927 03:20:21.227337 32459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:21.246093 32459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:21.248394 32459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:21.248555 32459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:21.248842 32459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:21.249285 32510 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:20:21.251345 32459 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:20:21.251405 32459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:20:21.251446 32459 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:20:21.252065 32459 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:20:21.252122 32459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:21.252192 32510 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap starting.
I20260927 03:20:21.259307 32459 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:36979
I20260927 03:20:21.259454 32617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:36979 every 8 connection(s)
I20260927 03:20:21.259711 32459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:21.261720 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32459
I20260927 03:20:21.262022 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:37637
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=41425
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:21.263520 32510 log.cc:824] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:21.266335 32510 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:21.266669 32510 tablet_bootstrap.cc:492] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Bootstrap complete.
I20260927 03:20:21.267880 32510 ts_tablet_manager.cc:1397] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260927 03:20:21.269342 32510 raft_consensus.cc:348] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:21.269745 32510 raft_consensus.cc:729] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106fce8bcc4d4f9ca94ad9fac0a756eb, State: Initialized, Role: FOLLOWER
I20260927 03:20:21.269991 32510 consensus_queue.cc:260] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:21.270301 32510 ts_tablet_manager.cc:1428] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260927 03:20:21.346212 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:21.346457 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:21.346496 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:21.347632 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:21.347697 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:21.347744 32621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:21.349319 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=41425
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:21.349596 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:21.349871 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:21.354837 32631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.354866 32629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.354848 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:21.355033 32621 server_base.cc:1034] running on GCE node
I20260927 03:20:21.355281 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:21.355532 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:21.356650 32621 hybrid_clock.cc:648] HybridClock initialized: now 1790479221356623 us; error 26 us; skew 500 ppm
Sep 27 03:20:21 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:21.359740 32621 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:21.360178 32621 webserver.cc:460] Webserver started at http://127.17.52.130:41425/ using document root <none> and password file <none>
I20260927 03:20:21.360412 32621 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:21.360459 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:21.362213 32621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:21.362900 32638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.363107 32621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:21.363201 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783"
format_stamp: "Formatted at 2026-09-27 03:20:19 on dist-test-slave-7j43"
server_key: "308c50f6f74d313ccd926334f3918c6d"
server_key_iv: "c76ac045823b6465793cf4408d00033d"
server_key_version: "encryptionkey@0"
I20260927 03:20:21.363620 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:21.382791 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:21.384821 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:21.384934 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:21.385159 32621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:21.385504 32621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:21.385565 32621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.385607 32621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:21.385632 32621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.391657 32621 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:37637
I20260927 03:20:21.391687 32751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:37637 every 8 connection(s)
I20260927 03:20:21.392136 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:20:21.399521 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32621
I20260927 03:20:21.399772 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:33569
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:33681
--webserver_port=42377
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:21.417356 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:20:21.260488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:39843)
0927 03:20:21.260653 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.260696 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:21.261122 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:21.417140 (+156018us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":119,"threads_started":1}
W20260927 03:20:21.417610 32618 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:33681: Client connection negotiation failed: client connection to 127.17.52.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:20:21.417802 32754 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:20:21.392881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:35663)
0927 03:20:21.393041 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.393074 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:21.393525 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:21.417143 (+ 23618us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260927 03:20:21.417953 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.417803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:55455)
0927 03:20:21.417868 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.417921 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:20:21.418102 32752 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:33681: Client connection negotiation failed: client connection to 127.17.52.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:20:21.418366 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.418261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:50733)
0927 03:20:21.418301 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.418323 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:20:21.418428 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.418311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:58487)
0927 03:20:21.418362 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.418403 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:20:21.418514 32618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:20:21.418792 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.418618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:53713)
0927 03:20:21.418711 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.418734 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:20:21.418900 32752 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:20:21.482225 32755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:21.482426 32755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:21.482450 32755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:21.483518 32755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:21.483551 32755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:21.483575 32755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:21.484982 32755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:33569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=42377
--tserver_master_addrs=127.17.52.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:21.485242 32755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:21.485491 32755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:21.490216 32760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.490226 32763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:21.490231 32761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:21.490473 32755 server_base.cc:1034] running on GCE node
I20260927 03:20:21.490638 32755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:21.490845 32755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:21.491982 32755 hybrid_clock.cc:648] HybridClock initialized: now 1790479221491959 us; error 40 us; skew 500 ppm
Sep 27 03:20:21 dist-test-slave-7j43 krb5kdc[31321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:21.494437 32755 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:21.494874 32755 webserver.cc:460] Webserver started at http://127.17.52.131:42377/ using document root <none> and password file <none>
I20260927 03:20:21.495049 32755 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:21.495090 32755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:21.496346 32755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:21.496996 302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.497165 32755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.497267 32755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "7918afab54d542da9394952f38842aa9"
format_stamp: "Formatted at 2026-09-27 03:20:20 on dist-test-slave-7j43"
server_key: "6e2e6548670bc4101a5bbe3b62193316"
server_key_iv: "dd42d6877b671c88728446f58cb1a36e"
server_key_version: "encryptionkey@0"
I20260927 03:20:21.497601 32755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:21.509407 32755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:21.511561 32755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:21.511672 32755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:21.511921 32755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:21.512202 32755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:21.512233 32755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.512255 32755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:21.512307 32755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:21.517889 32755 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:33569
I20260927 03:20:21.517925 415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:33569 every 8 connection(s)
I20260927 03:20:21.518266 32755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:20:21.519268 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.518907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:55477)
0927 03:20:21.519068 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.519138 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
W20260927 03:20:21.519702 416 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:33681: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
I20260927 03:20:21.519980 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.519891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:36893)
0927 03:20:21.519937 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.519964 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:20:21.520238 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:21.520166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:46795)
0927 03:20:21.520204 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:21.520225 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:20:21.520349 416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:20:21.525604 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 32755
I20260927 03:20:22.419054 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.418828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:47839)
0927 03:20:22.418952 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:22.418987 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260927 03:20:22.419369 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.419188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:39337)
0927 03:20:22.419275 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:22.419320 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":50,"threads_started":1}
I20260927 03:20:22.465090 422 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:22.465211 422 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:22.465512 422 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
Sep 27 03:20:22 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.131@KRBTEST.COM
Sep 27 03:20:22 dist-test-slave-7j43 krb5kdc[31321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479221, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.130@KRBTEST.COM
I20260927 03:20:22.477396 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.465858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:51149 (local address 127.17.52.130:37637)
0927 03:20:22.466021 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:22.466027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:22.466038 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:20:22.466164 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:22.466166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:22.466190 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:22.466297 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:22.467026 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.468494 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.469952 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.470580 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.470696 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:22.473978 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:22.473982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:22.473982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:22.473992 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:22.475588 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:22.476060 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:22.476063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:22.476063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:22.476169 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:22.476390 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:22.476391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:22.476391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:22.476719 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:22.476799 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:22.477140 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 03:20:22.477269 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260927 03:20:22.477445 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.465770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:33569 (local address 127.17.52.129:33383)
0927 03:20:22.466019 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:22.466028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:22.466073 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:22.466834 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:22.466835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:22.467002 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:22.467085 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:22.467088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.468796 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.468797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:22.469868 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:22.469870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.470675 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.470677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:22.470809 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:22.471442 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:22.471446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:22.473381 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:22.475522 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:22.475528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:22.475530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:22.475871 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:22.476170 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:22.476170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:22.476170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:22.476316 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:22.476794 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:22.476799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:22.476952 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 03:20:22.477068 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:20:22.477335 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":119,"thread_start_us":64,"threads_started":1}
I20260927 03:20:22.477555 425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.465907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:33383 (local address 127.17.52.131:33569)
0927 03:20:22.466079 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:22.466090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:22.466114 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:20:22.466608 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:22.466610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:22.466694 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:22.466800 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:22.467241 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.468706 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.469953 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.470580 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.470696 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:22.473499 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:22.473503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:22.473503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:22.473513 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:22.475418 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:22.475955 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:22.475957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:22.475957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:22.476094 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:22.476390 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:22.476391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:22.476391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:22.476719 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:22.476799 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:22.477140 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 03:20:22.477365 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":53,"threads_started":1}
I20260927 03:20:22.478118 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.465651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:37637 (local address 127.17.52.129:51149)
0927 03:20:22.465710 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:22.465724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:22.465814 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:22.466349 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:22.466352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:22.466637 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:22.466894 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:22.466901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.468629 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.468630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:22.469868 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:22.469870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:22.470677 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:22.470678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:22.470815 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:22.471428 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:22.471439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:22.473763 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:22.475720 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:22.475721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:22.475721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:22.475968 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:22.476249 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:22.476250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:22.476250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:22.476315 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:22.476799 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:22.476800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:22.476952 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 03:20:22.477068 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:20:22.477275 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":35}
W20260927 03:20:22.478384 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:22.478919 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:22.478973 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:22.479111 422 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:22.520882 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:22.520694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:57177)
0927 03:20:22.520808 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:22.520848 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 03:20:23.419691 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:23.419445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:60817)
0927 03:20:23.419569 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:23.419601 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260927 03:20:23.419981 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:23.419755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:46991)
0927 03:20:23.419882 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:23.419919 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 03:20:23.521461 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:23.521275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:34539)
0927 03:20:23.521394 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:23.521428 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260927 03:20:24.420516 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:24.420102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:33399)
0927 03:20:24.420398 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:24.420447 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":55,"threads_started":1}
I20260927 03:20:24.420732 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:24.420479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:37807)
0927 03:20:24.420589 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:24.420671 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":52,"threads_started":1}
I20260927 03:20:24.522030 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:24.521849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:35375)
0927 03:20:24.521962 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:24.521997 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260927 03:20:24.643404 433 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:24.643491 433 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:24.643603 433 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
W20260927 03:20:24.644019 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:24.644127 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:24.644161 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:24.644246 433 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:25.421346 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:25.421122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:54797)
0927 03:20:25.421252 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:25.421294 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260927 03:20:25.421353 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:25.421122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:54761)
0927 03:20:25.421280 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:25.421313 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260927 03:20:25.522805 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:25.522612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:40807)
0927 03:20:25.522727 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:25.522764 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 03:20:26.422026 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:26.421829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:59711)
0927 03:20:26.421953 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:26.421983 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260927 03:20:26.422065 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:26.421854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:46225)
0927 03:20:26.421986 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:26.422024 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260927 03:20:26.523461 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:26.523264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:45059)
0927 03:20:26.523387 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:26.523422 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 03:20:26.886736 440 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:26.886833 440 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:26.886956 440 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
W20260927 03:20:26.887388 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:26.887491 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:26.887511 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:26.887598 440 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:27.422618 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:27.422412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:46493)
0927 03:20:27.422544 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:27.422573 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260927 03:20:27.422750 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:27.422513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:41789)
0927 03:20:27.422647 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:27.422685 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260927 03:20:27.524075 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:27.523880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:34939)
0927 03:20:27.524005 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:27.524037 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260927 03:20:28.423343 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:28.423122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:59007)
0927 03:20:28.423257 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:28.423298 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260927 03:20:28.423360 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:28.423160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:57973)
0927 03:20:28.423269 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:28.423319 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260927 03:20:28.524765 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:28.524510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:53163)
0927 03:20:28.524651 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:28.524703 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":106,"threads_started":1}
I20260927 03:20:29.423998 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:29.423752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:33461)
0927 03:20:29.423905 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:29.423944 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260927 03:20:29.424005 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:29.423771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:54655)
0927 03:20:29.423930 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:29.423966 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260927 03:20:29.525434 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:29.525216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:33731)
0927 03:20:29.525360 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:29.525394 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260927 03:20:29.694072 450 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:29.694199 450 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:29.694418 450 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
W20260927 03:20:29.694821 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:29.694893 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:29.694908 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:29.694973 450 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:30.424707 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:30.424425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:43351)
0927 03:20:30.424561 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:30.424667 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260927 03:20:30.424705 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:30.424425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:41339)
0927 03:20:30.424631 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:30.424660 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260927 03:20:30.526289 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:30.526035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:52261)
0927 03:20:30.526203 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:30.526245 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260927 03:20:31.425309 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:31.425102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:51065)
0927 03:20:31.425230 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:31.425261 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260927 03:20:31.425464 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:31.425253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:59101)
0927 03:20:31.425367 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:31.425397 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260927 03:20:31.526973 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:31.526753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:51943)
0927 03:20:31.526888 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:31.526929 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260927 03:20:32.426133 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:32.425846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:53013)
0927 03:20:32.426001 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:32.426071 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260927 03:20:32.426184 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:32.425960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:55421)
0927 03:20:32.426069 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:32.426128 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260927 03:20:32.527683 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:32.527456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:54557)
0927 03:20:32.527599 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:32.527637 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260927 03:20:33.137428 460 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:33.137519 460 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:33.137636 460 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
W20260927 03:20:33.138053 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:33.138137 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:33.138190 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:33.138284 460 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:33.427013 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:33.426814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:52831)
0927 03:20:33.426941 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:33.426970 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260927 03:20:33.427013 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:33.426794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:58979)
0927 03:20:33.426929 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:33.426965 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260927 03:20:33.528566 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:33.528282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:60201)
0927 03:20:33.528438 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:33.528490 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260927 03:20:34.427757 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:34.427529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:39551)
0927 03:20:34.427683 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:34.427716 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260927 03:20:34.427747 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:34.427529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:60293)
0927 03:20:34.427663 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:34.427701 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 03:20:34.529443 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:34.529200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:58557)
0927 03:20:34.529339 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:34.529397 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 03:20:35.428472 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:35.428247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:58009)
0927 03:20:35.428388 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:35.428424 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260927 03:20:35.428472 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:35.428243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:39171)
0927 03:20:35.428393 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:35.428426 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260927 03:20:35.530133 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:35.529914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:56325)
0927 03:20:35.530037 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:35.530075 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260927 03:20:36.276760 470 raft_consensus.cc:482] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:36.276868 470 raft_consensus.cc:504] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fd9d5bf137a45cb9ad0a805ae0b8783" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 37637 } } peers { permanent_uuid: "7918afab54d542da9394952f38842aa9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 33569 } } peers { permanent_uuid: "106fce8bcc4d4f9ca94ad9fac0a756eb" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 36979 } }
I20260927 03:20:36.277168 470 leader_election.cc:288] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637), 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569)
W20260927 03:20:36.277395 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fd9d5bf137a45cb9ad0a805ae0b8783 (127.17.52.130:37637): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
W20260927 03:20:36.277498 32505 leader_election.cc:341] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7918afab54d542da9394952f38842aa9 (127.17.52.131:33569): Not found: Tablet not found: 18682963dd434b03ac44fbc8faebc538
I20260927 03:20:36.277532 32505 leader_election.cc:302] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106fce8bcc4d4f9ca94ad9fac0a756eb; no voters: 6fd9d5bf137a45cb9ad0a805ae0b8783, 7918afab54d542da9394952f38842aa9
I20260927 03:20:36.277623 470 raft_consensus.cc:2726] T 18682963dd434b03ac44fbc8faebc538 P 106fce8bcc4d4f9ca94ad9fac0a756eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:20:36.429221 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:36.428983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.130:44575)
0927 03:20:36.429134 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:36.429163 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260927 03:20:36.429243 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:36.428983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.129:56043)
0927 03:20:36.429144 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:36.429195 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260927 03:20:36.529721 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 32459
I20260927 03:20:36.531275 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:36.531081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:33681 (local address 127.17.52.131:56297)
0927 03:20:36.531214 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:36.531238 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":46,"threads_started":1}
I20260927 03:20:36.534577 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 32621
I20260927 03:20:36.538759 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 32755
2026-09-27T03:20:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18235 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:20:36 dist-test-slave-7j43 krb5kdc[476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:20:36 dist-test-slave-7j43 krb5kdc[476](info): set up 2 sockets
Sep 27 03:20:36 dist-test-slave-7j43 krb5kdc[476](info): commencing operation
krb5kdc: starting...
W20260927 03:20:37.569088 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:20:37 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:20:37.681252 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:37.761675 489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:37.761897 489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:37.761931 489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:37.763077 489 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:20:37.763144 489 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:20:37.763166 489 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:20:37.763185 489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:37.763217 489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:37.763243 489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:37.763275 489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:37.763301 489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:37.764806 489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:37.765071 489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:37.765297 489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:37.770364 494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:37.770406 497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:37.770423 489 server_base.cc:1034] running on GCE node
W20260927 03:20:37.770371 495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:37.770921 489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:37.771222 489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:37.772388 489 hybrid_clock.cc:648] HybridClock initialized: now 1790479237772363 us; error 37 us; skew 500 ppm
Sep 27 03:20:37 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479237, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:37.774895 489 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:37.775305 489 webserver.cc:460] Webserver started at http://127.17.52.190:41137/ using document root <none> and password file <none>
I20260927 03:20:37.775523 489 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:37.775589 489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:37.775735 489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:37.776865 489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "5aa2b6db29174f4a848b84f9abce92f0"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "e6ed5bbc0510ab1c5f5d6f3c7a1220d2"
server_key_iv: "370c4a6352595656865f29a45d84bdf7"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.777215 489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "5aa2b6db29174f4a848b84f9abce92f0"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "e6ed5bbc0510ab1c5f5d6f3c7a1220d2"
server_key_iv: "370c4a6352595656865f29a45d84bdf7"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.778508 489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:37.779217 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:37.779392 489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:37.779496 489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "5aa2b6db29174f4a848b84f9abce92f0"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "e6ed5bbc0510ab1c5f5d6f3c7a1220d2"
server_key_iv: "370c4a6352595656865f29a45d84bdf7"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.779569 489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:37.798717 489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:37.801108 489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:37.801275 489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:37.805016 489 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:41085
I20260927 03:20:37.805088 556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:41085 every 8 connection(s)
I20260927 03:20:37.805436 489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:37.806277 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:37.808435 557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap starting.
I20260927 03:20:37.809059 557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:37.809306 557 log.cc:824] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:37.810030 557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: No bootstrap required, opened a new log
I20260927 03:20:37.811607 557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:37.811759 557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:37.811784 557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Initialized, Role: FOLLOWER
I20260927 03:20:37.811928 557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:37.812016 557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:37.812039 557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:37.812068 557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:37.812552 557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:37.812704 557 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa2b6db29174f4a848b84f9abce92f0; no voters:
I20260927 03:20:37.812837 557 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:20:37.812881 560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:37.813045 560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Running, Role: LEADER
I20260927 03:20:37.813117 557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:37.813182 560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:37.813757 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:20:37.813848 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:37.813809 562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa2b6db29174f4a848b84f9abce92f0. Latest consensus state: current_term: 1 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:20:37.813898 562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:37.814061 569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:37.814694 569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:37.815462 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 489
I20260927 03:20:37.815580 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:20:37.815791 17618 external_mini_cluster.cc:1428] Setting key ccc771962f3a81367577451650380af8
I20260927 03:20:37.816246 569 catalog_manager.cc:1347] Generated new cluster ID: 0161a9cac7ee4d238519258bd1e5c2fe
I20260927 03:20:37.816301 569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:20:37 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:37.826457 569 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:20:37.827023 569 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:37.827641 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:37.816440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.17.52.190:41085)
0927 03:20:37.816635 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:37.816644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:37.816676 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:20:37.816802 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:37.816809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:37.816861 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:37.816984 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:37.817738 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.819110 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.820216 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.820830 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.820926 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:37.823855 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:37.823860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:37.823860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:37.823875 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:37.825910 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:37.826273 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:37.826276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:37.826277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:37.826391 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:37.826641 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:37.826642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:37.826642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:37.826929 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:37.827070 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:37.827245 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:20:37.827404 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260927 03:20:37.830714 569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Generated new TSK 0
I20260927 03:20:37.830904 569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:20:37.861039 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:37.941649 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:37.941848 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:37.941874 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:37.942939 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:37.942974 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:37.942999 583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:37.944336 583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:37.944501 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:37.944685 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:37.949465 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:37.949465 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:37.949541 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:37.949579 583 server_base.cc:1034] running on GCE node
I20260927 03:20:37.949869 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:37.950136 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:37.951313 583 hybrid_clock.cc:648] HybridClock initialized: now 1790479237951296 us; error 31 us; skew 500 ppm
Sep 27 03:20:37 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479237, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:37.953881 583 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:37.954319 583 webserver.cc:460] Webserver started at http://127.17.52.129:42713/ using document root <none> and password file <none>
I20260927 03:20:37.954680 583 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:37.954749 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:37.954936 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:37.956151 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "c276dc95948a4948bd62648ae4a70732"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "bb5eba3017a336abbe23e5e422092ebc"
server_key_iv: "64755d6b97103e1774b14462ff9f706a"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.956512 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "c276dc95948a4948bd62648ae4a70732"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "bb5eba3017a336abbe23e5e422092ebc"
server_key_iv: "64755d6b97103e1774b14462ff9f706a"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.957829 583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:37.958516 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:37.958693 583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:20:37.958761 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "c276dc95948a4948bd62648ae4a70732"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "bb5eba3017a336abbe23e5e422092ebc"
server_key_iv: "64755d6b97103e1774b14462ff9f706a"
server_key_version: "encryptionkey@0"
I20260927 03:20:37.958810 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:37.972771 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:37.974969 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:37.975078 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:37.975244 583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:37.975544 583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:37.975580 583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:37.975601 583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:37.975615 583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:37.981401 583 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46687
I20260927 03:20:37.981462 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46687 every 8 connection(s)
I20260927 03:20:37.981781 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:37.985723 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 583
I20260927 03:20:37.985839 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:20:37.986047 17618 external_mini_cluster.cc:1428] Setting key 9174901a3d891c819409cfce08230496
Sep 27 03:20:37 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479237, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:37.991321 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:37.982493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:54489 (local address 127.17.52.190:41085)
0927 03:20:37.982541 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:37.982543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:37.982755 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:20:37.983229 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:37.983230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:37.983239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:37.983285 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:37.984159 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.984723 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.985654 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.985982 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.986049 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:37.988400 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:37.988401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:37.988401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:37.988407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:37.989916 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:37.990380 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:37.990381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:37.990381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:37.990432 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:37.990712 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:37.990713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:37.990713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:37.990806 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:37.990851 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:37.991172 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 03:20:37.991235 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:20:37.991621 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:37.982473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:54489)
0927 03:20:37.982626 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:37.982672 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:37.983128 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:37.983353 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:37.983356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:37.983892 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:37.984083 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:37.984087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.984797 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.984798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:37.985577 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:37.985579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:37.986073 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:37.986074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:37.986251 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:37.986841 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:37.986848 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:37.988300 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:37.989998 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:37.990004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:37.990006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:37.990303 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:37.990494 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:37.990495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:37.990495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:37.990643 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:37.990865 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:37.990871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:37.991032 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 03:20:37.991153 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:20:37.991343 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260927 03:20:37.992190 712 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:37.992276 712 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:37.992471 712 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:37.992980 521 ts_manager.cc:194] Registered new tserver with Master: c276dc95948a4948bd62648ae4a70732 (127.17.52.129:46687)
I20260927 03:20:37.993552 521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:54489
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:20:38.009009 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:38.089290 717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:38.089510 717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:38.089534 717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:38.090628 717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:38.090663 717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:38.090688 717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:38.092010 717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:38.092180 717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:38.092340 717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:38.097147 725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:38.097199 722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:38.097208 717 server_base.cc:1034] running on GCE node
W20260927 03:20:38.097164 723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:38.097554 717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:38.097802 717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:38.098964 717 hybrid_clock.cc:648] HybridClock initialized: now 1790479238098950 us; error 26 us; skew 500 ppm
Sep 27 03:20:38 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479238, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:38.101562 717 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:38.101975 717 webserver.cc:460] Webserver started at http://127.17.52.130:33519/ using document root <none> and password file <none>
I20260927 03:20:38.102190 717 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:38.102259 717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:38.102413 717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:38.103468 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "8d0426552a2f47b49dbd3484e66b9d35"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "8ca4e4104f03e05368551416ba20ae85"
server_key_iv: "5f0bee98ed9aecd5e1e4efcb323f7284"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.103823 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "8d0426552a2f47b49dbd3484e66b9d35"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "8ca4e4104f03e05368551416ba20ae85"
server_key_iv: "5f0bee98ed9aecd5e1e4efcb323f7284"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.105170 717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:20:38.105836 732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.105996 717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:38.106097 717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "8d0426552a2f47b49dbd3484e66b9d35"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "8ca4e4104f03e05368551416ba20ae85"
server_key_iv: "5f0bee98ed9aecd5e1e4efcb323f7284"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.106168 717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:38.137537 717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:38.139982 717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:38.140144 717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:38.140396 717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:38.140758 717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:38.140817 717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.140857 717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:38.140888 717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.146759 717 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:44665
I20260927 03:20:38.146816 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:44665 every 8 connection(s)
I20260927 03:20:38.147104 717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:20:38 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479238, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:38.154953 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 717
I20260927 03:20:38.155053 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:20:38.155259 17618 external_mini_cluster.cc:1428] Setting key a68ece3a6529ca79427f3e3c900a84af
I20260927 03:20:38.155881 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.147782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:45853 (local address 127.17.52.190:41085)
0927 03:20:38.147853 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.147856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.148067 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:20:38.148515 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.148519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.148531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.148629 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:38.149515 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.150001 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.150926 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.151101 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.151151 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.153058 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:38.153060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:38.153060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:38.153066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:38.154456 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.154833 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.154834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.154835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.154899 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.155136 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.155137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.155138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.155277 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:38.155356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.155691 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.155780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:20:38.156067 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.147768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:45853)
0927 03:20:38.147950 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.147984 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.148437 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.148699 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.148702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.149233 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:38.149436 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.149440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.150074 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.150075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.150854 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.150856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.151169 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.151170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.151264 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.151654 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:38.151662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:38.152980 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:38.154520 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:38.154525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:38.154526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:38.154728 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:38.154967 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:38.154968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:38.154968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:38.155057 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:38.155354 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:38.155359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:38.155533 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.155651 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.155801 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260927 03:20:38.156486 846 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:38.156610 846 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:38.156760 846 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:38.157080 521 ts_manager.cc:194] Registered new tserver with Master: 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
I20260927 03:20:38.157451 521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:45853
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:20:38.177423 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:38.257694 851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:38.257884 851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:38.257905 851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:38.259028 851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:38.259069 851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:38.259095 851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:38.260504 851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:38.260757 851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:38.260982 851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:20:38.265829 851 server_base.cc:1034] running on GCE node
W20260927 03:20:38.265803 859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:38.265866 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:38.265799 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:38.266243 851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:38.266497 851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:38.267659 851 hybrid_clock.cc:648] HybridClock initialized: now 1790479238267655 us; error 37 us; skew 500 ppm
Sep 27 03:20:38 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479238, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:38.270149 851 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:38.270632 851 webserver.cc:460] Webserver started at http://127.17.52.131:35539/ using document root <none> and password file <none>
I20260927 03:20:38.270855 851 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:38.270925 851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:38.271083 851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:20:38.272116 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "b823738c45aa4fecbbc0b3c34f283a25"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "f091f66a2c23d72e08ef35a9619d1275"
server_key_iv: "1df24bd543c8d7319c0c95281d18c3e3"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.272501 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "b823738c45aa4fecbbc0b3c34f283a25"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "f091f66a2c23d72e08ef35a9619d1275"
server_key_iv: "1df24bd543c8d7319c0c95281d18c3e3"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.273908 851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:38.274627 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.274793 851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:20:38.274893 851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "b823738c45aa4fecbbc0b3c34f283a25"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "f091f66a2c23d72e08ef35a9619d1275"
server_key_iv: "1df24bd543c8d7319c0c95281d18c3e3"
server_key_version: "encryptionkey@0"
I20260927 03:20:38.274969 851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:38.285629 851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:38.288308 851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:38.288475 851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:38.288770 851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:38.289088 851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:38.289141 851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.289182 851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:38.289211 851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:38.295006 851 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:41233
I20260927 03:20:38.295055 979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:41233 every 8 connection(s)
I20260927 03:20:38.295363 851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:20:38 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479238, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:38.301781 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 851
I20260927 03:20:38.301908 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:20:38.302142 17618 external_mini_cluster.cc:1428] Setting key dabbdc400609fd0422c51f834bb7385f
I20260927 03:20:38.303855 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.296012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:40157 (local address 127.17.52.190:41085)
0927 03:20:38.296079 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.296081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.296252 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:20:38.296778 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.296779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.296788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.296851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:38.297728 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.298228 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.299145 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.299324 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.299373 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.301316 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:38.301317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:38.301318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:38.301323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:38.302792 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.303212 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.303213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.303213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.303263 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.303425 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.303425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.303426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.303511 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:38.303555 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.303727 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.303782 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:20:38.304031 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.295970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:40157)
0927 03:20:38.296125 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.296166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.296695 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.296910 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.296913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.297468 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:38.297659 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.297663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.298315 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.298316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.299076 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.299077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.299392 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.299393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.299486 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.299890 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:38.299897 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:38.301238 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:38.302874 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:38.302880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:38.302882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:38.303153 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:38.303303 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:38.303304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:38.303304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:38.303391 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:38.303544 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:38.303546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:38.303645 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.303716 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.303827 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 03:20:38.304432 980 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:38.304507 980 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:38.304703 980 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:38.305033 521 ts_manager.cc:194] Registered new tserver with Master: b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233)
I20260927 03:20:38.305400 521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:40157
I20260927 03:20:38.313871 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:38.321305 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.315379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.17.52.190:41085)
0927 03:20:38.315448 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.315451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.315557 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:20:38.315635 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.315636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.315648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.315712 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:38.316295 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.316837 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.317609 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.317787 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.317862 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.319017 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:38.319019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:38.319019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:38.319026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:38.320100 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.320453 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.320454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.320455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.320510 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:38.320735 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:38.320736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:38.320737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:38.320889 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:38.320997 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.321129 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.321203 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:20:38.322824 521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:20:38.324414 521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:20:38.330643 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.326464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46687 (local address 127.0.0.1:52822)
0927 03:20:38.326653 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.326695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.326765 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.326995 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.326999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.327001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:38.327069 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.327071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328195 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.328196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.329470 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.329473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330318 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.330409 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330426 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.330491 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.330545 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260927 03:20:38.330739 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.326825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.17.52.131:41233)
0927 03:20:38.327014 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.327018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.327027 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:20:38.327063 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.327065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.327082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.327294 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:38.327367 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328433 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.329729 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330318 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330415 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330470 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.330601 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.330642 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260927 03:20:38.331043 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.326581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.17.52.129:46687)
0927 03:20:38.326766 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.326771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.326784 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:20:38.326871 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.326874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.326892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.326977 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:38.327152 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328093 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.329562 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330228 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330300 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330355 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.330903 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.330969 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260927 03:20:38.331081 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.327086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.17.52.130:44665)
0927 03:20:38.327244 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.327248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.327315 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:20:38.327382 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.327385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.327401 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.327472 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:38.327826 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328810 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.329965 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330675 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330735 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330784 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.330956 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.330997 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260927 03:20:38.331049 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.326694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44665 (local address 127.0.0.1:38800)
0927 03:20:38.327250 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.327260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.327318 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.327662 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.327662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.327665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:38.327701 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.327702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328903 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.328904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.329879 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.329882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330815 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.330872 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330876 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.330927 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.330970 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":47,"threads_started":1}
I20260927 03:20:38.331519 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.326496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:41233 (local address 127.0.0.1:41366)
0927 03:20:38.326742 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.326751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.326792 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.327179 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.327180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.327182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:38.327225 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.327227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.328537 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.328538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.329619 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.329621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.330443 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.330444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.330517 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.330521 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.330581 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.330623 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":7,"thread_start_us":66,"threads_started":1}
I20260927 03:20:38.331967 780 tablet_service.cc:1467] Processing CreateTablet for tablet 71e40c1b702245b3a8536569898d52f0 (DEFAULT_TABLE table=test-table [id=17d5de67bb854660b5f86f9a03a11374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:38.331983 646 tablet_service.cc:1467] Processing CreateTablet for tablet 71e40c1b702245b3a8536569898d52f0 (DEFAULT_TABLE table=test-table [id=17d5de67bb854660b5f86f9a03a11374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:38.332243 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e40c1b702245b3a8536569898d52f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:38.332244 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e40c1b702245b3a8536569898d52f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:38.334811 995 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Bootstrap starting.
I20260927 03:20:38.335301 994 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Bootstrap starting.
I20260927 03:20:38.335511 995 tablet_bootstrap.cc:654] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:38.335813 914 tablet_service.cc:1467] Processing CreateTablet for tablet 71e40c1b702245b3a8536569898d52f0 (DEFAULT_TABLE table=test-table [id=17d5de67bb854660b5f86f9a03a11374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:38.335986 994 tablet_bootstrap.cc:654] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:38.336136 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e40c1b702245b3a8536569898d52f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:38.336138 995 log.cc:824] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:38.336521 994 log.cc:824] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:38.336956 995 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: No bootstrap required, opened a new log
I20260927 03:20:38.337041 995 ts_tablet_manager.cc:1397] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:38.337342 994 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: No bootstrap required, opened a new log
I20260927 03:20:38.337409 994 ts_tablet_manager.cc:1397] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:20:38.339151 998 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Bootstrap starting.
I20260927 03:20:38.339100 995 raft_consensus.cc:348] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.339388 995 raft_consensus.cc:374] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:38.339321 994 raft_consensus.cc:348] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.339449 995 raft_consensus.cc:729] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Initialized, Role: FOLLOWER
I20260927 03:20:38.339485 994 raft_consensus.cc:374] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:38.339545 994 raft_consensus.cc:729] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0426552a2f47b49dbd3484e66b9d35, State: Initialized, Role: FOLLOWER
I20260927 03:20:38.339591 995 consensus_queue.cc:260] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.339691 994 consensus_queue.cc:260] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.339849 998 tablet_bootstrap.cc:654] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:38.339938 995 ts_tablet_manager.cc:1428] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:20:38.339958 994 ts_tablet_manager.cc:1428] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:20:38.340016 846 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
I20260927 03:20:38.340337 712 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
I20260927 03:20:38.340428 998 log.cc:824] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:38.341152 998 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: No bootstrap required, opened a new log
I20260927 03:20:38.341215 998 ts_tablet_manager.cc:1397] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:38.342705 998 raft_consensus.cc:348] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.342831 998 raft_consensus.cc:374] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:38.342852 998 raft_consensus.cc:729] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b823738c45aa4fecbbc0b3c34f283a25, State: Initialized, Role: FOLLOWER
I20260927 03:20:38.342988 998 consensus_queue.cc:260] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.343245 998 ts_tablet_manager.cc:1428] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:20:38.343272 980 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
W20260927 03:20:38.397641 847 tablet.cc:2369] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:20:38.482514 713 tablet.cc:2369] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:38.485258 999 raft_consensus.cc:482] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:38.485360 999 raft_consensus.cc:504] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.485762 999 leader_election.cc:288] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665), b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233)
I20260927 03:20:38.489723 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.486026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:57553 (local address 127.17.52.131:41233)
0927 03:20:38.486074 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.486076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.486090 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:20:38.486120 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.486121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.486137 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.486223 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:38.486433 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.487183 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.488769 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.489349 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.489443 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.489451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.489616 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.489653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:20:38.489723 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.486047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:49757 (local address 127.17.52.130:44665)
0927 03:20:38.486087 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.486089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.486174 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:20:38.486215 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.486217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.486229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.486279 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:38.486434 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.487179 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.488768 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.489369 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.489443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.489451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.489616 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.489653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:20:38.489760 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.485940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:41233 (local address 127.17.52.129:57553)
0927 03:20:38.486015 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.486025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.486067 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.486248 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.486249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.486250 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:38.486341 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.486344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.487265 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.487266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.488679 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.488682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.489456 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.489457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.489538 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.489540 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.489597 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.489650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":101}
I20260927 03:20:38.489768 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.485941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44665 (local address 127.17.52.129:49757)
0927 03:20:38.486101 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.486111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:38.486168 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:38.486292 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:38.486293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:38.486294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:38.486366 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.486368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.487265 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.487266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.488679 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:38.488682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.489456 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.489457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:38.489535 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.489539 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:20:38.489597 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:20:38.489650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":93,"thread_start_us":88,"threads_started":1}
I20260927 03:20:38.490144 934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e40c1b702245b3a8536569898d52f0" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b823738c45aa4fecbbc0b3c34f283a25" is_pre_election: true
I20260927 03:20:38.490275 800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e40c1b702245b3a8536569898d52f0" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0426552a2f47b49dbd3484e66b9d35" is_pre_election: true
I20260927 03:20:38.490326 934 raft_consensus.cc:2454] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 0.
I20260927 03:20:38.490411 800 raft_consensus.cc:2454] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 0.
I20260927 03:20:38.490547 599 leader_election.cc:302] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b823738c45aa4fecbbc0b3c34f283a25, c276dc95948a4948bd62648ae4a70732; no voters:
I20260927 03:20:38.490685 999 raft_consensus.cc:2781] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:38.490741 999 raft_consensus.cc:482] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:38.490758 999 raft_consensus.cc:3037] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:38.491423 999 raft_consensus.cc:504] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.491559 999 leader_election.cc:288] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 election: Requested vote from peers 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665), b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233)
I20260927 03:20:38.491704 934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e40c1b702245b3a8536569898d52f0" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b823738c45aa4fecbbc0b3c34f283a25"
I20260927 03:20:38.491734 800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e40c1b702245b3a8536569898d52f0" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0426552a2f47b49dbd3484e66b9d35"
I20260927 03:20:38.491781 934 raft_consensus.cc:3037] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:38.491822 800 raft_consensus.cc:3037] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:38.492491 800 raft_consensus.cc:2454] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 1.
I20260927 03:20:38.492553 934 raft_consensus.cc:2454] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 1.
I20260927 03:20:38.492728 601 leader_election.cc:302] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0426552a2f47b49dbd3484e66b9d35, c276dc95948a4948bd62648ae4a70732; no voters:
I20260927 03:20:38.492887 999 raft_consensus.cc:2781] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:38.493054 999 raft_consensus.cc:686] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 LEADER]: Becoming Leader. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Running, Role: LEADER
I20260927 03:20:38.493157 999 consensus_queue.cc:237] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:38.493947 520 catalog_manager.cc:5495] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 reported cstate change: term changed from 0 to 1, leader changed from <none> to c276dc95948a4948bd62648ae4a70732 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "c276dc95948a4948bd62648ae4a70732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20260927 03:20:38.510421 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.508198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.17.52.129:46687)
0927 03:20:38.508251 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:38.508253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:38.508261 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:20:38.508291 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:38.508292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:38.508319 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:38.508378 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:38.508471 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.509022 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.509747 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:38.509932 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:38.509978 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:38.510092 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:38.510185 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:38.510244 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:38.510278 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:20:38.510294 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:20:38.512027 800 raft_consensus.cc:1261] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Refusing update from remote peer c276dc95948a4948bd62648ae4a70732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:38.512368 934 raft_consensus.cc:1261] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Refusing update from remote peer c276dc95948a4948bd62648ae4a70732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:38.512388 1004 consensus_queue.cc:1035] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:38.512643 1004 consensus_queue.cc:1035] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:38.516866 1009 mvcc.cc:205] Tried to move back new op lower bound from 7333802960942952448 to 7333802960868528128. Current Snapshot: MvccSnapshot[applied={T|T < 7333802960942952448 or (T in {7333802960942952448})}]
I20260927 03:20:38.517177 520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:38.517262 520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:38.518006 520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Sending DeleteTablet for 3 replicas of tablet 71e40c1b702245b3a8536569898d52f0
I20260927 03:20:38.518323 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 583
I20260927 03:20:38.518342 914 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e40c1b702245b3a8536569898d52f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:38 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:41366
I20260927 03:20:38.518368 646 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e40c1b702245b3a8536569898d52f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:38 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:52822
I20260927 03:20:38.518868 780 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e40c1b702245b3a8536569898d52f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:38 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:38800
I20260927 03:20:38.519078 1017 tablet_replica.cc:330] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: stopping tablet replica
I20260927 03:20:38.519169 1017 raft_consensus.cc:2229] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:38.519235 1017 raft_consensus.cc:2258] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:38.519506 1016 tablet_replica.cc:330] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: stopping tablet replica
I20260927 03:20:38.519594 1016 raft_consensus.cc:2229] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:38.519599 1017 ts_tablet_manager.cc:1905] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:38.519682 1016 raft_consensus.cc:2258] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:38.520093 1016 ts_tablet_manager.cc:1905] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:38.521222 1017 ts_tablet_manager.cc:1918] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:38.521335 1017 log.cc:1196] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/71e40c1b702245b3a8536569898d52f0
I20260927 03:20:38.521651 1017 ts_tablet_manager.cc:1939] T 71e40c1b702245b3a8536569898d52f0 P 8d0426552a2f47b49dbd3484e66b9d35: Deleting consensus metadata
I20260927 03:20:38.522018 507 catalog_manager.cc:4841] TS 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): tablet 71e40c1b702245b3a8536569898d52f0 (table test-table [id=17d5de67bb854660b5f86f9a03a11374]) successfully deleted
I20260927 03:20:38.522113 1016 ts_tablet_manager.cc:1918] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:38.522184 1016 log.cc:1196] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/71e40c1b702245b3a8536569898d52f0
I20260927 03:20:38.522442 1016 ts_tablet_manager.cc:1939] T 71e40c1b702245b3a8536569898d52f0 P b823738c45aa4fecbbc0b3c34f283a25: Deleting consensus metadata
I20260927 03:20:38.522815 505 catalog_manager.cc:4841] TS b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): tablet 71e40c1b702245b3a8536569898d52f0 (table test-table [id=17d5de67bb854660b5f86f9a03a11374]) successfully deleted
W20260927 03:20:38.524887 507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:46687 (error 108)
I20260927 03:20:38.525085 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 717
I20260927 03:20:38.525345 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:38.525229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46687 (local address 127.0.0.1:52838)
0927 03:20:38.525275 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:38.525306 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:20:38.525444 507 catalog_manager.cc:4584] TS c276dc95948a4948bd62648ae4a70732 (127.17.52.129:46687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71e40c1b702245b3a8536569898d52f0: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46687: connect: Connection refused (error 111)
I20260927 03:20:38.530597 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 851
I20260927 03:20:38.534984 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 489
I20260927 03:20:58.541725 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--webserver_interface=127.17.52.190
--webserver_port=41137
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:20:58.622681 1019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:58.622896 1019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:58.622918 1019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:58.624100 1019 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:20:58.624143 1019 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:20:58.624157 1019 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:20:58.624168 1019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:20:58.624231 1019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:58.624249 1019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:58.624275 1019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:20:58.624305 1019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:20:58.625809 1019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:58.626072 1019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:58.626322 1019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:58.634760 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:58.634783 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:58.635042 1019 server_base.cc:1034] running on GCE node
W20260927 03:20:58.635170 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:58.635443 1019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:58.635798 1019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:58.636962 1019 hybrid_clock.cc:648] HybridClock initialized: now 1790479258636959 us; error 36 us; skew 500 ppm
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:58.639482 1019 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:20:58.639864 1019 webserver.cc:460] Webserver started at http://127.17.52.190:41137/ using document root <none> and password file <none>
I20260927 03:20:58.640057 1019 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:58.640110 1019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:58.641517 1019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:58.642120 1035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:58.642310 1019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:20:58.642395 1019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "5aa2b6db29174f4a848b84f9abce92f0"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "e6ed5bbc0510ab1c5f5d6f3c7a1220d2"
server_key_iv: "370c4a6352595656865f29a45d84bdf7"
server_key_version: "encryptionkey@0"
I20260927 03:20:58.642802 1019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:58.657646 1019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:58.659533 1019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:58.659641 1019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:58.663470 1019 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:41085
I20260927 03:20:58.663527 1087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:41085 every 8 connection(s)
I20260927 03:20:58.663908 1019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:20:58.665442 1088 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:20:58.666102 1088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap starting.
I20260927 03:20:58.666855 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1019
I20260927 03:20:58.667133 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:46687
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=42713
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:58.675117 1088 log.cc:824] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:58.677073 1088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:58.677385 1088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap complete.
I20260927 03:20:58.679603 1088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:58.679842 1088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Initialized, Role: FOLLOWER
I20260927 03:20:58.680040 1088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:58.680135 1088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:20:58.680188 1088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:20:58.680234 1088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:20:58.681254 1088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:58.681376 1088 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa2b6db29174f4a848b84f9abce92f0; no voters:
I20260927 03:20:58.681559 1088 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:20:58.681600 1092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:20:58.681720 1092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 LEADER]: Becoming Leader. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Running, Role: LEADER
I20260927 03:20:58.681825 1088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:20:58.681880 1092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:20:58.682247 1093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:20:58.682334 1093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:58.682621 1094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa2b6db29174f4a848b84f9abce92f0. Latest consensus state: current_term: 2 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:20:58.682672 1094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:20:58.682855 1100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:20:58.683631 1100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:20:58.683841 1100 catalog_manager.cc:1259] Loaded cluster ID: 0161a9cac7ee4d238519258bd1e5c2fe
I20260927 03:20:58.684011 1100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:20:58.685194 1100 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:20:58.685366 1100 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Loaded TSK: 0
I20260927 03:20:58.703794 1100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Generated new TSK 1
I20260927 03:20:58.703894 1100 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:20:58.749780 1090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:58.750007 1090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:58.750046 1090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:58.751199 1090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:58.751264 1090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:58.751309 1090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:20:58.752874 1090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:46687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=42713
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:58.753126 1090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:58.753345 1090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:58.759120 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:58.759169 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:58.759322 1090 server_base.cc:1034] running on GCE node
W20260927 03:20:58.759158 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:58.759589 1090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:58.759869 1090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:58.761044 1090 hybrid_clock.cc:648] HybridClock initialized: now 1790479258761017 us; error 38 us; skew 500 ppm
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:58.763772 1090 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:20:58.764340 1090 webserver.cc:460] Webserver started at http://127.17.52.129:42713/ using document root <none> and password file <none>
I20260927 03:20:58.764663 1090 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:58.764758 1090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:58.766256 1090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:58.767015 1126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:58.767221 1090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:58.767319 1090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "c276dc95948a4948bd62648ae4a70732"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "bb5eba3017a336abbe23e5e422092ebc"
server_key_iv: "64755d6b97103e1774b14462ff9f706a"
server_key_version: "encryptionkey@0"
I20260927 03:20:58.767690 1090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:58.779088 1090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:58.781328 1090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:58.781493 1090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:58.781771 1090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:58.782326 1133 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:20:58.784298 1090 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:20:58.784368 1090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:20:58.784415 1090 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:20:58.785146 1090 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:20:58.785208 1090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:58.785265 1133 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Bootstrap starting.
I20260927 03:20:58.793275 1090 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46687
I20260927 03:20:58.793779 1090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:20:58.794297 1240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46687 every 8 connection(s)
I20260927 03:20:58.797088 1133 log.cc:824] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:58.799345 1133 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:20:58.799690 1133 tablet_bootstrap.cc:492] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Bootstrap complete.
I20260927 03:20:58.800918 1133 ts_tablet_manager.cc:1397] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260927 03:20:58.802389 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1090
I20260927 03:20:58.802299 1133 raft_consensus.cc:348] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:58.802616 1133 raft_consensus.cc:729] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Initialized, Role: FOLLOWER
I20260927 03:20:58.802673 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:44665
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=33519
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:20:58.802781 1133 consensus_queue.cc:260] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } }
I20260927 03:20:58.803020 1133 ts_tablet_manager.cc:1428] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:58.811441 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.794580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:36923)
0927 03:20:58.799536 (+ 4956us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:58.799583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:58.800197 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:58.800728 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:58.800732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:58.801089 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:58.801402 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.801410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.803199 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.803200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.804280 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.804283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.805368 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.805369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.805472 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.806049 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:58.806058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:58.807569 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:58.809687 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.809694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.809696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.809939 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.810251 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.810252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.810252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.810374 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.810776 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:58.810783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:58.810923 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:20:58.811041 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:20:58.811229 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4908,"thread_start_us":85,"threads_started":1}
I20260927 03:20:58.811456 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.794702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:36923 (local address 127.17.52.190:41085)
0927 03:20:58.794887 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:58.794896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:58.799726 (+ 4830us) server_negotiation.cc:407] Connection header received
0927 03:20:58.800427 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:58.800434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:58.800495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:58.800689 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:58.801538 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.803049 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.804395 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.805160 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.805249 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.807694 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:58.807698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:58.807698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:58.807718 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:58.809551 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.810047 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.810050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.810051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.810151 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.810451 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.810452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.810453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.810690 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:58.810782 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:58.811099 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 03:20:58.811229 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260927 03:20:58.812644 1241 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:58.812744 1241 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:58.812925 1241 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:58.813696 1052 ts_manager.cc:194] Registered new tserver with Master: c276dc95948a4948bd62648ae4a70732 (127.17.52.129:46687)
I20260927 03:20:58.815016 1052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:36923
I20260927 03:20:58.816031 1241 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.129@KRBTEST.COM
I20260927 03:20:58.823742 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.814321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46687 (local address 127.0.0.1:54296)
0927 03:20:58.814734 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:58.814771 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:58.814832 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:58.815168 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:58.815173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:58.815543 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:58.816292 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.816296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.817556 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.817557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.818183 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.818185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.818923 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.818924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.819016 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.819501 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:58.819505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:58.820809 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:58.822559 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.822566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.822567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.822777 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.823004 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.823005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.823006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.823064 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.823362 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:58.823368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:58.823419 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:20:58.823519 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 03:20:58.823636 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":102,"threads_started":1}
I20260927 03:20:58.823742 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.814578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.17.52.129:46687)
0927 03:20:58.814950 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:58.814954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:58.814964 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:20:58.814995 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:58.814997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:58.815011 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:58.815076 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:58.816395 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.817403 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.818293 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.818803 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.818918 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.820899 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:58.820907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:58.820907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:58.820914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:58.822454 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.822874 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.822877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.822877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.822936 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.823143 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.823144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.823144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.823286 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:58.823369 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:58.823556 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:20:58.823647 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":71,"threads_started":1}
I20260927 03:20:58.824501 1159 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e40c1b702245b3a8536569898d52f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:38 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:54296
I20260927 03:20:58.825045 1250 tablet_replica.cc:330] stopping tablet replica
I20260927 03:20:58.825170 1250 raft_consensus.cc:2229] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:58.825239 1250 raft_consensus.cc:2258] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:58.825531 1250 ts_tablet_manager.cc:1905] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:58.827395 1250 ts_tablet_manager.cc:1918] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:58.827476 1250 log.cc:1196] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal/wals/71e40c1b702245b3a8536569898d52f0
I20260927 03:20:58.827807 1250 ts_tablet_manager.cc:1939] T 71e40c1b702245b3a8536569898d52f0 P c276dc95948a4948bd62648ae4a70732: Deleting consensus metadata
I20260927 03:20:58.828347 1038 catalog_manager.cc:4841] TS c276dc95948a4948bd62648ae4a70732 (127.17.52.129:46687): tablet 71e40c1b702245b3a8536569898d52f0 (table test-table [id=17d5de67bb854660b5f86f9a03a11374]) successfully deleted
W20260927 03:20:58.891350 1247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:58.891542 1247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:58.891566 1247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:58.892737 1247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:58.892778 1247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:58.892804 1247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:20:58.894207 1247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:44665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=33519
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:58.894435 1247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:58.894665 1247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:58.899467 1256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:58.899477 1258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:58.899542 1255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:58.899931 1247 server_base.cc:1034] running on GCE node
I20260927 03:20:58.900142 1247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:58.900393 1247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:58.901551 1247 hybrid_clock.cc:648] HybridClock initialized: now 1790479258901518 us; error 50 us; skew 500 ppm
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:58.903891 1247 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:20:58.904362 1247 webserver.cc:460] Webserver started at http://127.17.52.130:33519/ using document root <none> and password file <none>
I20260927 03:20:58.904662 1247 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:58.904744 1247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:58.906109 1247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:58.906810 1265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:58.906991 1247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:20:58.907080 1247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "8d0426552a2f47b49dbd3484e66b9d35"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "8ca4e4104f03e05368551416ba20ae85"
server_key_iv: "5f0bee98ed9aecd5e1e4efcb323f7284"
server_key_version: "encryptionkey@0"
I20260927 03:20:58.907440 1247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:58.926069 1247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:58.928139 1247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:58.928300 1247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:58.928540 1247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:58.928900 1247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:58.928956 1247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:58.928996 1247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:58.929024 1247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:58.934949 1247 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:44665
I20260927 03:20:58.934991 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:44665 every 8 connection(s)
I20260927 03:20:58.935328 1247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:20:58.939179 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1247
I20260927 03:20:58.939452 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:41233
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=35539
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:20:58 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479258, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:58.944878 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.935994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:36171)
0927 03:20:58.936127 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:58.936173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:58.936740 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:58.936993 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:58.936997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:58.937575 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:58.937759 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.937763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.938458 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.938459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.939314 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:58.939317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.939741 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.939742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:58.939879 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.940407 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:58.940415 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:58.941786 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:58.943563 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.943569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.943571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.943774 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.943981 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:58.943982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:58.943982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:58.944071 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:58.944355 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:58.944359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:58.944469 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 03:20:58.944549 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:20:58.944721 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260927 03:20:58.944913 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:58.936015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36171 (local address 127.17.52.190:41085)
0927 03:20:58.936065 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:58.936068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:58.936264 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:20:58.936849 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:58.936850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:58.936866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:58.936924 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:58.937863 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.938362 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.939386 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:58.939662 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:58.939733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:58.941880 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:58.941881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:58.941882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:58.941890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:58.943486 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.943849 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.943850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.943851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.943915 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:58.944133 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:58.944134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:58.944135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:58.944288 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:58.944363 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:58.944627 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 03:20:58.944776 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:20:58.945519 1379 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:58.945626 1379 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:58.945856 1379 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:58.946216 1052 ts_manager.cc:194] Registered new tserver with Master: 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
I20260927 03:20:58.946583 1052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:36171
W20260927 03:20:59.021859 1382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:20:59.022060 1382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:20:59.022084 1382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:20:59.023240 1382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:20:59.023281 1382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:20:59.023307 1382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:20:59.024788 1382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:41233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=35539
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:20:59.025030 1382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:20:59.025254 1382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:20:59.030078 1390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:59.030046 1388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:20:59.030047 1387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:20:59.030201 1382 server_base.cc:1034] running on GCE node
I20260927 03:20:59.030474 1382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:20:59.030669 1382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:20:59.031832 1382 hybrid_clock.cc:648] HybridClock initialized: now 1790479259031822 us; error 39 us; skew 500 ppm
Sep 27 03:20:59 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:20:59.037714 1382 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:20:59.038148 1382 webserver.cc:460] Webserver started at http://127.17.52.131:35539/ using document root <none> and password file <none>
I20260927 03:20:59.038388 1382 fs_manager.cc:359] Metadata directory not provided
I20260927 03:20:59.038468 1382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:20:59.039861 1382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:20:59.040463 1397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:59.040702 1382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:59.040800 1382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "b823738c45aa4fecbbc0b3c34f283a25"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "f091f66a2c23d72e08ef35a9619d1275"
server_key_iv: "1df24bd543c8d7319c0c95281d18c3e3"
server_key_version: "encryptionkey@0"
I20260927 03:20:59.041137 1382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:20:59.054664 1382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:20:59.056944 1382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:20:59.057101 1382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:20:59.057329 1382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:20:59.057642 1382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:20:59.057693 1382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:59.057734 1382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:20:59.057763 1382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:59.063701 1382 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:41233
I20260927 03:20:59.063745 1510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:41233 every 8 connection(s)
I20260927 03:20:59.064095 1382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:20:59.065142 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1382
Sep 27 03:20:59 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:20:59.072845 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.065561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.17.52.190:41085)
0927 03:20:59.065689 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.065691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.065878 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:20:59.065888 (+ 10us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.065888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.065894 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.065934 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:59.066729 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.067236 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.068020 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.068233 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.068313 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.069753 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:59.069755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:59.069755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:59.069762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:59.071570 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.071989 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.071990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.071991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.072053 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.072251 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.072253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.072253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.072403 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:59.072618 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.072660 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.072732 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260927 03:20:59.074043 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.064842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:50717 (local address 127.17.52.190:41085)
0927 03:20:59.064898 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.064900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.065141 (+ 241us) server_negotiation.cc:407] Connection header received
0927 03:20:59.065796 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.065797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.065807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.065848 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:59.067054 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.067517 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.068518 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.068761 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.068835 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.070943 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:59.070944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:59.070944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:59.070951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:59.072693 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.073158 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.073159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.073159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.073216 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.073452 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.073453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.073453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.073575 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:59.073635 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.073925 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.073986 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:20:59.074199 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.064851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:50717)
0927 03:20:59.065020 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.065054 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:59.065705 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:59.065950 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:59.065954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:59.066725 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:20:59.066961 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.066967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.067623 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.067624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.068415 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.068417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.068843 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.068845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.068987 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.069441 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:20:59.069448 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:20:59.070849 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:20:59.072766 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:59.072781 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:59.072783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:59.073089 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:59.073278 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:20:59.073278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:20:59.073279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:20:59.073401 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:20:59.073667 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:20:59.073672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:20:59.073801 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 03:20:59.073892 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:20:59.074006 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260927 03:20:59.074611 1511 heartbeater.cc:347] Connected to a master server at 127.17.52.190:41085
I20260927 03:20:59.074702 1511 heartbeater.cc:464] Registering TS with master...
I20260927 03:20:59.074847 1511 heartbeater.cc:511] Master 127.17.52.190:41085 requested a full tablet report, sending...
I20260927 03:20:59.075169 1052 ts_manager.cc:194] Registered new tserver with Master: b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233)
I20260927 03:20:59.075497 1052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:50717
I20260927 03:20:59.084441 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:20:59.087044 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.084842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.17.52.190:41085)
0927 03:20:59.084927 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.084929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.085003 (+ 74us) server_negotiation.cc:407] Connection header received
0927 03:20:59.085057 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.085058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.085091 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.085150 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:59.085255 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.085711 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.086444 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.086617 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.086664 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.086785 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:59.086846 (+ 61us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 03:20:59.086964 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44614: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 03:20:59.087105 1244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44614: token expired
W20260927 03:20:59.087173 984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.52.190:41085)
I20260927 03:20:59.092628 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.087434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.17.52.190:41085)
0927 03:20:59.087486 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.087487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.087494 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:20:59.087549 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.087550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.087557 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.087600 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:20:59.088108 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.088635 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.089272 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.089454 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.089503 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.090553 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:20:59.090554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:20:59.090554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:20:59.090560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:20:59.091731 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.092022 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.092023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.092023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.092073 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:20:59.092219 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:20:59.092219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:20:59.092219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:20:59.092310 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:20:59.092381 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.092458 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.092510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:20:59.093688 1052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:20:59.095244 1052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:20:59.097560 1159 tablet_service.cc:1467] Processing CreateTablet for tablet a605cd77373f483e82b887e2b648668c (DEFAULT_TABLE table=test-table [id=1708d1b53ec54624952560d1c133d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:59.097759 1159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605cd77373f483e82b887e2b648668c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:59.099014 1133 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Bootstrap starting.
I20260927 03:20:59.099335 1133 tablet_bootstrap.cc:654] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:59.099887 1133 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: No bootstrap required, opened a new log
I20260927 03:20:59.099946 1133 ts_tablet_manager.cc:1397] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:20:59.100065 1133 raft_consensus.cc:348] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.100129 1133 raft_consensus.cc:374] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:59.100171 1133 raft_consensus.cc:729] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Initialized, Role: FOLLOWER
I20260927 03:20:59.100216 1133 consensus_queue.cc:260] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.100303 1133 ts_tablet_manager.cc:1428] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:20:59.101157 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.097096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:41233 (local address 127.0.0.1:39288)
0927 03:20:59.097128 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.097138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:59.097175 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:59.097692 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:59.097694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:59.097695 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:59.097738 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.097739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.098754 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.098755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.099869 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.099872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.100900 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.100901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.100973 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.100990 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:20:59.101049 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:20:59.101093 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 03:20:59.101176 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.097268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.17.52.131:41233)
0927 03:20:59.097533 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.097537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.097544 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:20:59.097575 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.097577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.097591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.097676 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:59.097803 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.098672 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.100021 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.100807 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.100896 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.100954 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.101064 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.101099 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
I20260927 03:20:59.101876 1445 tablet_service.cc:1467] Processing CreateTablet for tablet a605cd77373f483e82b887e2b648668c (DEFAULT_TABLE table=test-table [id=1708d1b53ec54624952560d1c133d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:59.102217 1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605cd77373f483e82b887e2b648668c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:59.102409 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.097168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44665 (local address 127.0.0.1:56760)
0927 03:20:59.097334 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.097344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:59.097385 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:59.099224 (+ 1839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:59.099225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:59.099226 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:59.099267 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.099269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.100301 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.100302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.101321 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.101325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.102167 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.102168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.102215 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.102218 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:20:59.102291 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:20:59.102338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260927 03:20:59.102432 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.097264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.17.52.130:44665)
0927 03:20:59.099085 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.099090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.099100 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:20:59.099136 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.099138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.099154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.099228 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:59.099313 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.100242 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.101425 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.102085 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.102166 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.102219 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.102313 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.102356 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1787,"thread_start_us":116,"threads_started":1}
I20260927 03:20:59.103003 1313 tablet_service.cc:1467] Processing CreateTablet for tablet a605cd77373f483e82b887e2b648668c (DEFAULT_TABLE table=test-table [id=1708d1b53ec54624952560d1c133d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:20:59.103293 1313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605cd77373f483e82b887e2b648668c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:20:59.105019 1521 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Bootstrap starting.
I20260927 03:20:59.105573 1521 tablet_bootstrap.cc:654] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:59.105952 1522 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Bootstrap starting.
I20260927 03:20:59.106144 1521 log.cc:824] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:59.106590 1522 tablet_bootstrap.cc:654] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Neither blocks nor log segments found. Creating new log.
I20260927 03:20:59.106846 1521 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: No bootstrap required, opened a new log
I20260927 03:20:59.106923 1521 ts_tablet_manager.cc:1397] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:59.107004 1522 log.cc:824] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Log is configured to *not* fsync() on all Append() calls
I20260927 03:20:59.107609 1522 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: No bootstrap required, opened a new log
I20260927 03:20:59.107657 1522 ts_tablet_manager.cc:1397] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:20:59.108445 1521 raft_consensus.cc:348] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.108623 1521 raft_consensus.cc:374] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:59.108675 1521 raft_consensus.cc:729] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b823738c45aa4fecbbc0b3c34f283a25, State: Initialized, Role: FOLLOWER
I20260927 03:20:59.108798 1521 consensus_queue.cc:260] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.109091 1521 ts_tablet_manager.cc:1428] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:20:59.109150 1511 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
I20260927 03:20:59.109160 1522 raft_consensus.cc:348] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.109341 1522 raft_consensus.cc:374] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:20:59.109411 1522 raft_consensus.cc:729] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0426552a2f47b49dbd3484e66b9d35, State: Initialized, Role: FOLLOWER
I20260927 03:20:59.109525 1522 consensus_queue.cc:260] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.109783 1379 heartbeater.cc:503] Master 127.17.52.190:41085 was elected leader, sending a full tablet report...
I20260927 03:20:59.109807 1522 ts_tablet_manager.cc:1428] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 03:20:59.185832 1380 tablet.cc:2369] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:20:59.294986 1242 tablet.cc:2369] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:20:59.314733 1512 tablet.cc:2369] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:20:59.363420 1246 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:20:59.363555 1246 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.363886 1246 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
I20260927 03:20:59.368239 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.364279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:60751 (local address 127.17.52.131:41233)
0927 03:20:59.364324 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.364326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.364431 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:20:59.364507 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.364508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.364516 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.364606 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:59.364788 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.365558 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367229 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.367818 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367904 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.367913 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.368112 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.368160 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:20:59.368230 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.364199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50301 (local address 127.17.52.130:44665)
0927 03:20:59.364251 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.364253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.364358 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:20:59.364500 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.364501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.364516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.364606 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:20:59.364837 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.365570 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367229 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.367819 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367904 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.367913 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.368112 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.368161 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:20:59.368249 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.364104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:41233 (local address 127.17.52.129:60751)
0927 03:20:59.364358 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.364369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:59.364425 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:59.364610 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:59.364611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:59.364613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:59.364709 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.364712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.365664 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.365665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.367086 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.367090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.367913 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.367982 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.368007 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 03:20:59.368068 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:20:59.368128 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":99,"thread_start_us":65,"threads_started":1}
I20260927 03:20:59.368249 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.364104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44665 (local address 127.17.52.129:50301)
0927 03:20:59.364268 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.364280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:20:59.364391 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:20:59.364613 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:20:59.364613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:20:59.364615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:20:59.364758 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.364761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.365664 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.365665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.367086 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:20:59.367090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.367911 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.367913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:20:59.367986 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.368009 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 03:20:59.368068 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:20:59.368128 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":154,"thread_start_us":66,"threads_started":1}
I20260927 03:20:59.368673 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605cd77373f483e82b887e2b648668c" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b823738c45aa4fecbbc0b3c34f283a25" is_pre_election: true
I20260927 03:20:59.368835 1465 raft_consensus.cc:2454] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 0.
I20260927 03:20:59.368841 1333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605cd77373f483e82b887e2b648668c" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0426552a2f47b49dbd3484e66b9d35" is_pre_election: true
I20260927 03:20:59.368960 1333 raft_consensus.cc:2454] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 0.
I20260927 03:20:59.369092 1127 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b823738c45aa4fecbbc0b3c34f283a25, c276dc95948a4948bd62648ae4a70732; no voters:
I20260927 03:20:59.369225 1246 raft_consensus.cc:2781] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:20:59.369315 1246 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:20:59.369350 1246 raft_consensus.cc:3037] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:59.370050 1246 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.370215 1246 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 election: Requested vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
I20260927 03:20:59.370381 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605cd77373f483e82b887e2b648668c" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b823738c45aa4fecbbc0b3c34f283a25"
I20260927 03:20:59.370416 1333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605cd77373f483e82b887e2b648668c" candidate_uuid: "c276dc95948a4948bd62648ae4a70732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0426552a2f47b49dbd3484e66b9d35"
I20260927 03:20:59.370482 1465 raft_consensus.cc:3037] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:59.370488 1333 raft_consensus.cc:3037] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:20:59.371206 1465 raft_consensus.cc:2454] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 1.
I20260927 03:20:59.371309 1333 raft_consensus.cc:2454] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c276dc95948a4948bd62648ae4a70732 in term 1.
I20260927 03:20:59.371410 1127 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b823738c45aa4fecbbc0b3c34f283a25, c276dc95948a4948bd62648ae4a70732; no voters:
I20260927 03:20:59.371495 1246 raft_consensus.cc:2781] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:20:59.371624 1246 raft_consensus.cc:686] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 LEADER]: Becoming Leader. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Running, Role: LEADER
I20260927 03:20:59.371723 1246 consensus_queue.cc:237] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:20:59.372442 1051 catalog_manager.cc:5495] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 reported cstate change: term changed from 0 to 1, leader changed from <none> to c276dc95948a4948bd62648ae4a70732 (127.17.52.129). New cstate: current_term: 1 leader_uuid: "c276dc95948a4948bd62648ae4a70732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } health_report { overall_health: HEALTHY } } }
I20260927 03:20:59.386883 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.384680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.17.52.129:46687)
0927 03:20:59.384829 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 03:20:59.384831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:20:59.384836 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:20:59.384868 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:20:59.384868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:20:59.384885 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:20:59.384948 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:20:59.385071 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.385539 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.386216 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:20:59.386417 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:20:59.386466 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:20:59.386571 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:20:59.386666 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:20:59.386716 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:20:59.386750 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:20:59.386765 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260927 03:20:59.388464 1465 raft_consensus.cc:1261] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Refusing update from remote peer c276dc95948a4948bd62648ae4a70732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:59.388464 1333 raft_consensus.cc:1261] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Refusing update from remote peer c276dc95948a4948bd62648ae4a70732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:20:59.388895 1529 consensus_queue.cc:1035] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:59.389068 1246 consensus_queue.cc:1035] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:20:59.391907 1533 mvcc.cc:205] Tried to move back new op lower bound from 7333803046452338688 to 7333803046387183616. Current Snapshot: MvccSnapshot[applied={T|T < 7333803046452338688 or (T in {7333803046452338688})}]
I20260927 03:20:59.393723 1051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:59.393836 1051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:20:59.394562 1051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Sending DeleteTablet for 3 replicas of tablet a605cd77373f483e82b887e2b648668c
I20260927 03:20:59.394870 1313 tablet_service.cc:1514] Processing DeleteTablet for tablet a605cd77373f483e82b887e2b648668c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:59 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:56760
I20260927 03:20:59.394891 1159 tablet_service.cc:1514] Processing DeleteTablet for tablet a605cd77373f483e82b887e2b648668c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:59 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:54296
I20260927 03:20:59.394908 1445 tablet_service.cc:1514] Processing DeleteTablet for tablet a605cd77373f483e82b887e2b648668c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:20:59 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:39288
I20260927 03:20:59.395306 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1090
I20260927 03:20:59.395886 1543 tablet_replica.cc:330] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: stopping tablet replica
I20260927 03:20:59.396014 1543 raft_consensus.cc:2229] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:59.396116 1543 raft_consensus.cc:2258] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:59.396294 1541 tablet_replica.cc:330] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: stopping tablet replica
I20260927 03:20:59.396428 1541 raft_consensus.cc:2229] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:20:59.396534 1541 raft_consensus.cc:2258] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:20:59.396543 1543 ts_tablet_manager.cc:1905] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:59.396956 1541 ts_tablet_manager.cc:1905] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:20:59.398290 1543 ts_tablet_manager.cc:1918] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:59.398379 1543 log.cc:1196] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/a605cd77373f483e82b887e2b648668c
I20260927 03:20:59.398612 1541 ts_tablet_manager.cc:1918] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:20:59.398697 1541 log.cc:1196] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/a605cd77373f483e82b887e2b648668c
I20260927 03:20:59.398746 1543 ts_tablet_manager.cc:1939] T a605cd77373f483e82b887e2b648668c P 8d0426552a2f47b49dbd3484e66b9d35: Deleting consensus metadata
I20260927 03:20:59.399013 1541 ts_tablet_manager.cc:1939] T a605cd77373f483e82b887e2b648668c P b823738c45aa4fecbbc0b3c34f283a25: Deleting consensus metadata
I20260927 03:20:59.399190 1038 catalog_manager.cc:4841] TS 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): tablet a605cd77373f483e82b887e2b648668c (table test-table [id=1708d1b53ec54624952560d1c133d2f1]) successfully deleted
I20260927 03:20:59.399381 1036 catalog_manager.cc:4841] TS b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): tablet a605cd77373f483e82b887e2b648668c (table test-table [id=1708d1b53ec54624952560d1c133d2f1]) successfully deleted
W20260927 03:20:59.401712 1038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:46687 (error 108)
I20260927 03:20:59.402022 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1247
I20260927 03:20:59.402180 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:20:59.402050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46687 (local address 127.0.0.1:54316)
0927 03:20:59.402108 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:20:59.402144 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:20:59.402735 1038 catalog_manager.cc:4584] TS c276dc95948a4948bd62648ae4a70732 (127.17.52.129:46687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a605cd77373f483e82b887e2b648668c: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46687: connect: Connection refused (error 111)
I20260927 03:20:59.407481 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1382
I20260927 03:20:59.411875 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1019
I20260927 03:21:19.417163 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--webserver_interface=127.17.52.190
--webserver_port=41137
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:19.497452 1545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:19.497642 1545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:19.497663 1545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:19.498803 1545 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:21:19.498842 1545 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:21:19.498854 1545 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:21:19.498865 1545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:19.498919 1545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:19.498937 1545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:19.498961 1545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:19.498980 1545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:19.500429 1545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:19.500699 1545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:19.500929 1545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:19.505829 1550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.505878 1551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.505831 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:19.505848 1545 server_base.cc:1034] running on GCE node
I20260927 03:21:19.506237 1545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:19.506516 1545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:19.507680 1545 hybrid_clock.cc:648] HybridClock initialized: now 1790479279507665 us; error 26 us; skew 500 ppm
Sep 27 03:21:19 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:19.514287 1545 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:21:19.514744 1545 webserver.cc:460] Webserver started at http://127.17.52.190:41137/ using document root <none> and password file <none>
I20260927 03:21:19.514964 1545 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:19.515008 1545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:19.516314 1545 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:19.517006 1560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.517166 1545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.517256 1545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "5aa2b6db29174f4a848b84f9abce92f0"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "e6ed5bbc0510ab1c5f5d6f3c7a1220d2"
server_key_iv: "370c4a6352595656865f29a45d84bdf7"
server_key_version: "encryptionkey@0"
I20260927 03:21:19.517560 1545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:19.546226 1545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:19.548836 1545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:19.548944 1545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:19.552671 1545 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:41085
I20260927 03:21:19.552744 1612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:41085 every 8 connection(s)
I20260927 03:21:19.553103 1545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:19.554536 1613 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:21:19.555171 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap starting.
I20260927 03:21:19.562337 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1545
I20260927 03:21:19.562676 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:46687
--local_ip_for_outbound_sockets=127.17.52.129
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=42713
--webserver_interface=127.17.52.129
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:21:19.564380 1613 log.cc:824] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:19.566596 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:21:19.566887 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Bootstrap complete.
I20260927 03:21:19.569351 1613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:21:19.569584 1613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Initialized, Role: FOLLOWER
I20260927 03:21:19.569747 1613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:21:19.569804 1613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:19.569864 1613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:19.569907 1613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:21:19.570865 1613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:21:19.570947 1613 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa2b6db29174f4a848b84f9abce92f0; no voters:
I20260927 03:21:19.571115 1613 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:21:19.571182 1617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:21:19.571385 1613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:19.571420 1617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [term 3 LEADER]: Becoming Leader. State: Replica: 5aa2b6db29174f4a848b84f9abce92f0, State: Running, Role: LEADER
I20260927 03:21:19.571573 1617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } }
I20260927 03:21:19.571863 1617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa2b6db29174f4a848b84f9abce92f0. Latest consensus state: current_term: 3 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:21:19.571938 1617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:19.572379 1625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:19.572518 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5aa2b6db29174f4a848b84f9abce92f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2b6db29174f4a848b84f9abce92f0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 41085 } } }
I20260927 03:21:19.572664 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:19.573348 1625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:19.573472 1625 catalog_manager.cc:1259] Loaded cluster ID: 0161a9cac7ee4d238519258bd1e5c2fe
I20260927 03:21:19.573697 1625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:19.574942 1625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:19.575101 1625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Loaded TSK: 0
I20260927 03:21:19.575134 1625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aa2b6db29174f4a848b84f9abce92f0: Loaded TSK: 1
F20260927 03:21:19.575376 1625 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad67b859ad google::LogMessage::Fail() at ??:0
@ 0x7fad67b8888a google::LogMessage::SendToLog() at ??:0
@ 0x7fad67b854de google::LogMessage::Flush() at ??:0
@ 0x7fad67b86a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad6a2f2104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fad6a2efd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fad6fadf1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fad6fadf48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fad6fadf594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fad6faa5c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fad6faafad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fad69a530d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad69a48954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad6a8ac6db start_thread at ??:0
@ 0x7fad6668361f clone at ??:0
W20260927 03:21:19.665832 1615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:19.666044 1615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:19.666081 1615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:19.667191 1615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:19.667250 1615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:19.667294 1615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:19.668829 1615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:46687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=42713
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:19.669085 1615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:19.669303 1615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:19.674026 1655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.674114 1654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.674027 1657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:19.674717 1615 server_base.cc:1034] running on GCE node
I20260927 03:21:19.674896 1615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:19.675104 1615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:19.676256 1615 hybrid_clock.cc:648] HybridClock initialized: now 1790479279676244 us; error 27 us; skew 500 ppm
Sep 27 03:21:19 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:19.678787 1615 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:21:19.679268 1615 webserver.cc:460] Webserver started at http://127.17.52.129:42713/ using document root <none> and password file <none>
I20260927 03:21:19.679503 1615 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:19.679548 1615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:19.680831 1615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:19.681495 1664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.681676 1615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.681758 1615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "c276dc95948a4948bd62648ae4a70732"
format_stamp: "Formatted at 2026-09-27 03:20:37 on dist-test-slave-7j43"
server_key: "bb5eba3017a336abbe23e5e422092ebc"
server_key_iv: "64755d6b97103e1774b14462ff9f706a"
server_key_version: "encryptionkey@0"
I20260927 03:21:19.682075 1615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:19.696943 1615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:19.699024 1615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:19.699162 1615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:19.699388 1615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:19.699803 1671 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:21:19.701675 1615 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:21:19.701738 1615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:21:19.701782 1615 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:21:19.702359 1615 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:21:19.702414 1615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:21:19.702488 1671 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Bootstrap starting.
I20260927 03:21:19.709509 1615 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46687
I20260927 03:21:19.709872 1615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:19.712721 1778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46687 every 8 connection(s)
I20260927 03:21:19.713310 1671 log.cc:824] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:19.715919 1671 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:21:19.716219 1671 tablet_bootstrap.cc:492] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Bootstrap complete.
I20260927 03:21:19.717460 1671 ts_tablet_manager.cc:1397] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260927 03:21:19.718313 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1615
I20260927 03:21:19.718578 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:44665
--local_ip_for_outbound_sockets=127.17.52.130
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=33519
--webserver_interface=127.17.52.130
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:21:19.718832 1671 raft_consensus.cc:348] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:19.719147 1671 raft_consensus.cc:729] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c276dc95948a4948bd62648ae4a70732, State: Initialized, Role: FOLLOWER
I20260927 03:21:19.719283 1671 consensus_queue.cc:260] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:19.719545 1671 ts_tablet_manager.cc:1428] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260927 03:21:19.798925 1781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:21:19.710502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:42969)
0927 03:21:19.710674 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.710727 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:19.711151 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:19.798709 (+ 87558us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260927 03:21:19.799134 1779 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:41085 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:41085: Client connection negotiation failed: client connection to 127.17.52.190:41085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:21:19.799465 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.799344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:48555)
0927 03:21:19.799399 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.799445 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:21:19.799865 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.799772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:59327)
0927 03:21:19.799826 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.799848 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:21:19.799993 1779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:21:19.812831 1784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:19.813015 1784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:19.813038 1784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:19.814248 1784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:19.814286 1784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:19.814313 1784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:19.815822 1784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:44665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=33519
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:19.816093 1784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:19.816354 1784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:21:19.821305 1784 server_base.cc:1034] running on GCE node
W20260927 03:21:19.821288 1790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.821295 1792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.821278 1789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:19.821705 1784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:19.821966 1784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:19.823129 1784 hybrid_clock.cc:648] HybridClock initialized: now 1790479279823123 us; error 34 us; skew 500 ppm
Sep 27 03:21:19 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:19.825608 1784 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:21:19.826013 1784 webserver.cc:460] Webserver started at http://127.17.52.130:33519/ using document root <none> and password file <none>
I20260927 03:21:19.826242 1784 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:19.826311 1784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:19.827601 1784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:19.828205 1799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.828361 1784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:19.828459 1784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "8d0426552a2f47b49dbd3484e66b9d35"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "8ca4e4104f03e05368551416ba20ae85"
server_key_iv: "5f0bee98ed9aecd5e1e4efcb323f7284"
server_key_version: "encryptionkey@0"
I20260927 03:21:19.828843 1784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:19.839856 1784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:19.841634 1784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:19.841789 1784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:19.842000 1784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:19.842283 1784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:19.842344 1784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.842384 1784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:19.842414 1784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.847952 1784 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:44665
I20260927 03:21:19.848052 1912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:44665 every 8 connection(s)
I20260927 03:21:19.848362 1784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:19.849380 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.849034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:41643)
0927 03:21:19.849168 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.849234 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
W20260927 03:21:19.849776 1913 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:41085 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:41085: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
I20260927 03:21:19.850033 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.849940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:54647)
0927 03:21:19.849990 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.850016 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:21:19.850267 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.850211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:35741)
0927 03:21:19.850237 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.850255 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 03:21:19.850373 1913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:21:19.854295 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1784
I20260927 03:21:19.854525 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:41233
--local_ip_for_outbound_sockets=127.17.52.131
--tserver_master_addrs=127.17.52.190:41085
--webserver_port=35539
--webserver_interface=127.17.52.131
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:19.935755 1916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:19.935958 1916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:19.935981 1916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:19.937099 1916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:19.937139 1916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:19.937165 1916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:19.938597 1916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:41233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=35539
--tserver_master_addrs=127.17.52.190:41085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:19.938841 1916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:19.939095 1916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:19.944141 1922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:19.944141 1924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:19.944321 1916 server_base.cc:1034] running on GCE node
W20260927 03:21:19.944130 1921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:19.944645 1916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:19.944859 1916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:19.946036 1916 hybrid_clock.cc:648] HybridClock initialized: now 1790479279946024 us; error 28 us; skew 500 ppm
Sep 27 03:21:19 dist-test-slave-7j43 krb5kdc[476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:19.948488 1916 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:21:19.948905 1916 webserver.cc:460] Webserver started at http://127.17.52.131:35539/ using document root <none> and password file <none>
I20260927 03:21:19.949095 1916 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:19.949134 1916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:19.950335 1916 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:19.950949 1931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.951136 1916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:19.951231 1916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "b823738c45aa4fecbbc0b3c34f283a25"
format_stamp: "Formatted at 2026-09-27 03:20:38 on dist-test-slave-7j43"
server_key: "f091f66a2c23d72e08ef35a9619d1275"
server_key_iv: "1df24bd543c8d7319c0c95281d18c3e3"
server_key_version: "encryptionkey@0"
I20260927 03:21:19.951574 1916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:19.969868 1916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:19.972009 1916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:19.972163 1916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:19.972421 1916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:19.972826 1916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:19.972885 1916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.972931 1916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:19.972962 1916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:19.979256 1916 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:41233
I20260927 03:21:19.979363 2044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:41233 every 8 connection(s)
I20260927 03:21:19.979774 1916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:19.980890 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.980486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:33917)
0927 03:21:19.980669 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.980741 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":116,"threads_started":1}
W20260927 03:21:19.981317 2045 heartbeater.cc:644] Failed to heartbeat to 127.17.52.190:41085 (0 consecutive failures): Network error: Failed to ping master at 127.17.52.190:41085: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
I20260927 03:21:19.981623 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.981500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:40651)
0927 03:21:19.981568 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.981601 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 03:21:19.982043 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:19.981949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:35527)
0927 03:21:19.981997 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:19.982024 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:21:19.982208 2045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:21:19.989559 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 1916
I20260927 03:21:20.800663 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:20.800383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:49843)
0927 03:21:20.800508 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:20.800545 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 03:21:20.850854 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:20.850647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:53871)
0927 03:21:20.850771 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:20.850802 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260927 03:21:20.982789 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:20.982575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:33813)
0927 03:21:20.982695 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:20.982735 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260927 03:21:21.324843 2052 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:21.324966 2052 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:21.325269 2052 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
Sep 27 03:21:21 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.130@KRBTEST.COM
Sep 27 03:21:21 dist-test-slave-7j43 krb5kdc[476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479279, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.131@KRBTEST.COM
I20260927 03:21:21.336258 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.325423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:41233 (local address 127.17.52.129:46877)
0927 03:21:21.325579 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:21.325591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:21.325652 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:21.326624 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:21.326627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:21.327006 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:21.327211 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:21.327217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.328331 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.328332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:21.329171 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:21.329173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.329787 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:21.329905 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:21.330574 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:21.330578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:21.332621 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:21.334787 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:21.334788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:21.334789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:21.334997 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:21.335246 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:21.335246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:21.335247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:21.335309 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:21.335713 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:21.335713 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:21.335779 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 03:21:21.335905 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:21:21.336113 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":62,"thread_start_us":96,"threads_started":1}
I20260927 03:21:21.336258 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.325469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:44665 (local address 127.17.52.129:38873)
0927 03:21:21.325593 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:21.325605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:21.325697 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:21.326693 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:21.326694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:21.327004 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:21.327211 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:21.327217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.328325 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.328328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:21.329168 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:21.329170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.329783 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:21.329916 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:21.330517 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:21.330527 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:21.332147 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:21.334435 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:21.334443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:21.334444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:21.334687 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:21.334978 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:21.334979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:21.334979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:21.335127 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:21.335600 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:21.335606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:21.335779 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 03:21:21.335905 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:21:21.336167 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":145,"thread_start_us":64,"threads_started":1}
I20260927 03:21:21.336323 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.325530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:46877 (local address 127.17.52.131:41233)
0927 03:21:21.325707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:21.325720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:21.325754 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:21:21.326362 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:21.326365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:21.326406 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:21.326558 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:21.327383 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.328231 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329277 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.329683 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329820 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:21.332719 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:21.332722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:21.332722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:21.332733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:21.334678 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:21.335078 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:21.335080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:21.335080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:21.335172 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:21.335371 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:21.335372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:21.335372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:21.335646 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:21.335717 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:21.336045 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 03:21:21.336189 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":55,"threads_started":1}
I20260927 03:21:21.336302 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.325621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:38873 (local address 127.17.52.130:44665)
0927 03:21:21.325777 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:21.325784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:21.325810 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:21:21.326343 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:21.326346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:21.326407 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:21.326558 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:21.327334 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.328214 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329264 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:21.329682 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:21.329799 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:21.332248 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:21.332251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:21.332252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:21.332261 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:21.334337 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:21.334823 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:21.334826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:21.334826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:21.334908 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:21.335193 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:21.335194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:21.335194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:21.335527 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:21.335611 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:21.336045 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 03:21:21.336189 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":49,"threads_started":1}
W20260927 03:21:21.337327 1665 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
W20260927 03:21:21.337424 1667 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
I20260927 03:21:21.337524 1667 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c276dc95948a4948bd62648ae4a70732; no voters: 8d0426552a2f47b49dbd3484e66b9d35, b823738c45aa4fecbbc0b3c34f283a25
I20260927 03:21:21.337630 2052 raft_consensus.cc:2726] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:21:21.801321 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.801196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:33691)
0927 03:21:21.801247 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:21.801280 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:21:21.851449 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.851254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:46491)
0927 03:21:21.851374 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:21.851410 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
I20260927 03:21:21.983397 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:21.983203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:40597)
0927 03:21:21.983325 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:21.983361 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260927 03:21:22.801959 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:22.801740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:55391)
0927 03:21:22.801875 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:22.801911 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260927 03:21:22.841434 2060 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:22.841543 2060 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:22.841717 2060 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
W20260927 03:21:22.842109 1667 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
W20260927 03:21:22.842212 1665 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
I20260927 03:21:22.842295 1665 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c276dc95948a4948bd62648ae4a70732; no voters: 8d0426552a2f47b49dbd3484e66b9d35, b823738c45aa4fecbbc0b3c34f283a25
I20260927 03:21:22.842396 2060 raft_consensus.cc:2726] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:21:22.852078 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:22.851875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:38667)
0927 03:21:22.852007 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:22.852041 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 03:21:22.984184 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:22.983926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:50585)
0927 03:21:22.984095 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:22.984136 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260927 03:21:23.802699 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:23.802495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:46035)
0927 03:21:23.802614 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:23.802652 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260927 03:21:23.852730 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:23.852509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:47199)
0927 03:21:23.852654 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:23.852689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260927 03:21:23.984935 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:23.984701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:57271)
0927 03:21:23.984865 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:23.984896 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260927 03:21:24.490033 2066 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:24.490126 2066 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:24.490319 2066 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
W20260927 03:21:24.490679 1667 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
W20260927 03:21:24.490728 1665 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
I20260927 03:21:24.490756 1667 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c276dc95948a4948bd62648ae4a70732; no voters: 8d0426552a2f47b49dbd3484e66b9d35, b823738c45aa4fecbbc0b3c34f283a25
I20260927 03:21:24.490882 2066 raft_consensus.cc:2726] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:21:24.803400 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:24.803187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:37353)
0927 03:21:24.803314 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:24.803352 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260927 03:21:24.853377 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:24.853161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:34265)
0927 03:21:24.853284 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:24.853336 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 03:21:24.985510 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:24.985326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:57695)
0927 03:21:24.985436 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:24.985471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260927 03:21:25.804148 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:25.803924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:36189)
0927 03:21:25.804056 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:25.804096 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260927 03:21:25.854130 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:25.853953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:39145)
0927 03:21:25.854060 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:25.854089 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260927 03:21:25.986119 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:25.985913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:59569)
0927 03:21:25.986032 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:25.986074 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260927 03:21:26.804916 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:26.804727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:37311)
0927 03:21:26.804843 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:26.804870 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260927 03:21:26.854791 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:26.854558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:53267)
0927 03:21:26.854707 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:26.854749 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260927 03:21:26.986754 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:26.986550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:48673)
0927 03:21:26.986674 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:26.986713 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260927 03:21:27.805526 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:27.805327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:44435)
0927 03:21:27.805441 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:27.805479 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 03:21:27.855382 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:27.855207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:58047)
0927 03:21:27.855309 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:27.855344 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260927 03:21:27.987409 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:27.987193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:52475)
0927 03:21:27.987325 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:27.987365 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260927 03:21:28.690146 2079 raft_consensus.cc:482] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:28.690304 2079 raft_consensus.cc:504] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b823738c45aa4fecbbc0b3c34f283a25" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 41233 } } peers { permanent_uuid: "8d0426552a2f47b49dbd3484e66b9d35" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 44665 } } peers { permanent_uuid: "c276dc95948a4948bd62648ae4a70732" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46687 } }
I20260927 03:21:28.690487 2079 leader_election.cc:288] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233), 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665)
W20260927 03:21:28.690932 1667 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d0426552a2f47b49dbd3484e66b9d35 (127.17.52.130:44665): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
W20260927 03:21:28.691030 1665 leader_election.cc:341] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b823738c45aa4fecbbc0b3c34f283a25 (127.17.52.131:41233): Not found: Tablet not found: a605cd77373f483e82b887e2b648668c
I20260927 03:21:28.691107 1665 leader_election.cc:302] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c276dc95948a4948bd62648ae4a70732; no voters: 8d0426552a2f47b49dbd3484e66b9d35, b823738c45aa4fecbbc0b3c34f283a25
I20260927 03:21:28.691169 2079 raft_consensus.cc:2726] T a605cd77373f483e82b887e2b648668c P c276dc95948a4948bd62648ae4a70732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:21:28.806216 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:28.805993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:35393)
0927 03:21:28.806125 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:28.806168 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260927 03:21:28.856055 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:28.855803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:41149)
0927 03:21:28.855954 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:28.855995 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260927 03:21:28.988044 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:28.987812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:57893)
0927 03:21:28.987950 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:28.987986 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260927 03:21:29.807117 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:29.806815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:47985)
0927 03:21:29.807031 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:29.807069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
I20260927 03:21:29.856814 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:29.856527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:50031)
0927 03:21:29.856722 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:29.856765 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":117,"threads_started":1}
I20260927 03:21:29.988744 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:29.988488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:50009)
0927 03:21:29.988657 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:29.988697 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260927 03:21:30.807852 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:30.807610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:55291)
0927 03:21:30.807755 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:30.807800 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260927 03:21:30.857519 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:30.857284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:46145)
0927 03:21:30.857432 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:30.857473 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260927 03:21:30.989473 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:30.989234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:46703)
0927 03:21:30.989388 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:30.989426 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260927 03:21:31.808517 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:31.808295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:45069)
0927 03:21:31.808430 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:31.808468 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260927 03:21:31.858204 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:31.857999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:59317)
0927 03:21:31.858124 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:31.858158 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260927 03:21:31.990096 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:31.989898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:52253)
0927 03:21:31.990016 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:31.990052 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 03:21:32.809381 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:32.809143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:38187)
0927 03:21:32.809279 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:32.809321 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260927 03:21:32.858868 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:32.858657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:56301)
0927 03:21:32.858778 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:32.858816 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260927 03:21:32.990839 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:32.990600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:44979)
0927 03:21:32.990752 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:32.990793 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260927 03:21:33.810242 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:33.809964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:47945)
0927 03:21:33.810124 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:33.810192 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260927 03:21:33.859582 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:33.859349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:44105)
0927 03:21:33.859492 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:33.859538 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260927 03:21:33.991439 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:33.991250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:35157)
0927 03:21:33.991363 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:33.991397 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260927 03:21:34.811038 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:34.810831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.129:40847)
0927 03:21:34.810965 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:34.810992 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260927 03:21:34.860414 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:34.860179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.130:54165)
0927 03:21:34.860325 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:34.860366 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260927 03:21:34.992093 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:34.991849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:41085 (local address 127.17.52.131:52847)
0927 03:21:34.991984 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:34.992029 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.190:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260927 03:21:34.999388 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1615
I20260927 03:21:35.004257 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1784
I20260927 03:21:35.008700 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 1916
2026-09-27T03:21:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58470 ms)
[----------] 30 tests from SecurityITest (228912 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:35 dist-test-slave-7j43 krb5kdc[2103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:35 dist-test-slave-7j43 krb5kdc[2103](info): set up 2 sockets
Sep 27 03:21:35 dist-test-slave-7j43 krb5kdc[2103](info): commencing operation
krb5kdc: starting...
W20260927 03:21:36.036861 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:21:36.151530 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45061
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:36.231189 2116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:36.231391 2116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:36.231415 2116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:36.232622 2116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:36.232663 2116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:36.232678 2116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:36.232726 2116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:36.232755 2116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:36.234213 2116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:36.234465 2116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:36.234691 2116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:36.240008 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.240036 2124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:36.240039 2116 server_base.cc:1034] running on GCE node
W20260927 03:21:36.240008 2121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:36.240423 2116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:36.240726 2116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:36.241870 2116 hybrid_clock.cc:648] HybridClock initialized: now 1790479296241850 us; error 36 us; skew 500 ppm
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:36.244215 2116 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:21:36.244697 2116 webserver.cc:460] Webserver started at http://127.17.52.190:34395/ using document root <none> and password file <none>
I20260927 03:21:36.244936 2116 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:36.244982 2116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:36.245126 2116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:36.246161 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "9404160ca7d049b6a76f70c79428d7ae"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "14aefb98d7a39da0a184a17c961663d1"
server_key_iv: "cf4fde032052559a036ab6bd828ba3c0"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.246526 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "9404160ca7d049b6a76f70c79428d7ae"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "14aefb98d7a39da0a184a17c961663d1"
server_key_iv: "cf4fde032052559a036ab6bd828ba3c0"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.247820 2116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:36.248493 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.248654 2116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:36.248754 2116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "9404160ca7d049b6a76f70c79428d7ae"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "14aefb98d7a39da0a184a17c961663d1"
server_key_iv: "cf4fde032052559a036ab6bd828ba3c0"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.248840 2116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:36.263643 2116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:36.265568 2116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:36.265728 2116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:36.269600 2116 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:45061
I20260927 03:21:36.269651 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:45061 every 8 connection(s)
I20260927 03:21:36.270012 2116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:36.270880 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:36.273200 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: Bootstrap starting.
I20260927 03:21:36.273738 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:36.273983 2184 log.cc:824] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:36.274676 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: No bootstrap required, opened a new log
I20260927 03:21:36.275854 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2116
I20260927 03:21:36.275977 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:36.276170 17618 external_mini_cluster.cc:1428] Setting key 3e84d1b2fd89b78a8bae8b56bc3c49fb
I20260927 03:21:36.276427 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } }
I20260927 03:21:36.276639 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:36.276733 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9404160ca7d049b6a76f70c79428d7ae, State: Initialized, Role: FOLLOWER
I20260927 03:21:36.276888 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } }
I20260927 03:21:36.276989 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:36.277025 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:36.277077 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:36.278117 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } }
I20260927 03:21:36.278224 2184 leader_election.cc:302] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9404160ca7d049b6a76f70c79428d7ae; no voters:
I20260927 03:21:36.278391 2184 leader_election.cc:288] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:36.278446 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:36.278589 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [term 1 LEADER]: Becoming Leader. State: Replica: 9404160ca7d049b6a76f70c79428d7ae, State: Running, Role: LEADER
I20260927 03:21:36.278635 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:36.278708 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } }
I20260927 03:21:36.279119 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9404160ca7d049b6a76f70c79428d7ae. Latest consensus state: current_term: 1 leader_uuid: "9404160ca7d049b6a76f70c79428d7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } } }
I20260927 03:21:36.279254 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:36.279527 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:36.279976 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9404160ca7d049b6a76f70c79428d7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9404160ca7d049b6a76f70c79428d7ae" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45061 } } }
I20260927 03:21:36.280088 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:36.280157 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:36.282202 2198 catalog_manager.cc:1347] Generated new cluster ID: 14ab72c7f08341debc1cf500c32e9501
I20260927 03:21:36.282263 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:36.287986 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:36.288072 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.276760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.17.52.190:45061)
0927 03:21:36.276984 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.276993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.277025 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:21:36.277171 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.277179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.277238 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.277379 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:36.278007 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.279600 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.280838 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.281458 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.281946 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.284462 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:36.284465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:36.284466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:36.284477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:36.286391 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.286804 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.286807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.286808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.286921 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.287145 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.287146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.287146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.287463 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:36.287620 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.287714 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.287853 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260927 03:21:36.288738 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:36.295028 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: Generated new TSK 0
I20260927 03:21:36.295202 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:21:36.323388 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:36.403652 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:36.403896 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:36.403940 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:36.405109 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:36.405179 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:36.405224 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:36.406859 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:36.407124 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:36.407354 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:36.412871 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.412882 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.412889 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:36.413156 2210 server_base.cc:1034] running on GCE node
I20260927 03:21:36.413419 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:36.413671 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:36.414839 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790479296414843 us; error 46 us; skew 500 ppm
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:36.418001 2210 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:21:36.418612 2210 webserver.cc:460] Webserver started at http://127.17.52.129:34175/ using document root <none> and password file <none>
I20260927 03:21:36.418889 2210 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:36.418975 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:36.419117 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:36.420181 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "597dda84b38c4115acccca5f858cd37c"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "dd490d7c8574b27c965f04a3a27e037a"
server_key_iv: "94fa3f239c2fd5fca189839f83c08cff"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.420624 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "597dda84b38c4115acccca5f858cd37c"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "dd490d7c8574b27c965f04a3a27e037a"
server_key_iv: "94fa3f239c2fd5fca189839f83c08cff"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.421972 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:36.422667 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.422822 2210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:36.422909 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "597dda84b38c4115acccca5f858cd37c"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "dd490d7c8574b27c965f04a3a27e037a"
server_key_iv: "94fa3f239c2fd5fca189839f83c08cff"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.422971 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:36.430835 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:36.432709 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:36.432822 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:36.433002 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:36.433307 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:36.433346 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.433403 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:36.433429 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.439074 2210 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:43243
I20260927 03:21:36.439152 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:43243 every 8 connection(s)
I20260927 03:21:36.439474 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:36.441061 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2210
I20260927 03:21:36.441177 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:36.441380 17618 external_mini_cluster.cc:1428] Setting key f7632756af5e9856bc752e8988542950
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:36.449955 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.440187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45061 (local address 127.17.52.129:59349)
0927 03:21:36.440364 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.440406 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.441034 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.441364 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.441369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.442160 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:36.442460 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.442467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.443150 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.443152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.444145 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.444148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.444676 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.444678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.444794 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.445369 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:36.445377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:36.446733 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:36.448559 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.448565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.448567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.448869 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.449079 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.449080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.449080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.449173 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.449477 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:36.449481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:36.449593 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.449673 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.449789 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260927 03:21:36.449952 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.440211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:59349 (local address 127.17.52.190:45061)
0927 03:21:36.440276 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.440278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.440493 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:21:36.441162 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.441163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.441179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.441254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:36.442556 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.443052 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.444249 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.444585 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.444651 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.446828 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:36.446830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:36.446831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:36.446840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:36.448483 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.448949 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.448950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.448951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.449009 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.449243 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.449244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.449244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.449407 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:36.449484 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.449730 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.449828 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
I20260927 03:21:36.450443 2339 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45061
I20260927 03:21:36.450544 2339 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:36.450738 2339 heartbeater.cc:511] Master 127.17.52.190:45061 requested a full tablet report, sending...
I20260927 03:21:36.451366 2148 ts_manager.cc:194] Registered new tserver with Master: 597dda84b38c4115acccca5f858cd37c (127.17.52.129:43243)
I20260927 03:21:36.452046 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:59349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:21:36.466969 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:36.547904 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:36.548102 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:36.548125 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:36.549216 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:36.549256 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:36.549283 2344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:36.550670 2344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:36.550853 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:36.551033 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:36.556075 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.556143 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.556083 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:36.556316 2344 server_base.cc:1034] running on GCE node
I20260927 03:21:36.556515 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:36.556785 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:36.557926 2344 hybrid_clock.cc:648] HybridClock initialized: now 1790479296557900 us; error 44 us; skew 500 ppm
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:36.560302 2344 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:21:36.560820 2344 webserver.cc:460] Webserver started at http://127.17.52.130:39673/ using document root <none> and password file <none>
I20260927 03:21:36.561058 2344 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:36.561103 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:36.561249 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:36.562323 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "01eec0e02aff44528de55fae4683661d"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "d397b266ceabf81fadd9e567a6422cf2"
server_key_iv: "25bda73e788a326562951de627ae858c"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.562687 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "01eec0e02aff44528de55fae4683661d"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "d397b266ceabf81fadd9e567a6422cf2"
server_key_iv: "25bda73e788a326562951de627ae858c"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.563961 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:36.564715 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.564875 2344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:36.564971 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "01eec0e02aff44528de55fae4683661d"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "d397b266ceabf81fadd9e567a6422cf2"
server_key_iv: "25bda73e788a326562951de627ae858c"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.565058 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:36.581508 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:36.583282 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:36.583448 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:36.583671 2344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:36.584030 2344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:36.584085 2344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.584124 2344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:36.584153 2344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.589936 2344 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:36141
I20260927 03:21:36.589977 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:36141 every 8 connection(s)
I20260927 03:21:36.590310 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:36.591740 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2344
I20260927 03:21:36.591873 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:36.592073 17618 external_mini_cluster.cc:1428] Setting key f9bd984ce481d23587f3cf4d8c6806d8
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:36.606055 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.591074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45061 (local address 127.17.52.130:46461)
0927 03:21:36.591234 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.591276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.591761 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.592030 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.592034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.592840 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:36.593058 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.593063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.593745 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.593747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.595090 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.595093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.595453 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.595454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.595586 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.599839 (+ 4253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:36.599847 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:36.602831 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:36.604684 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.604689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.604691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.604917 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.605128 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.605129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.605129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.605222 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.605555 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:36.605560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:36.605673 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.605774 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.605893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260927 03:21:36.606055 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.591118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:46461 (local address 127.17.52.190:45061)
0927 03:21:36.591166 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.591168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.591370 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:21:36.591883 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.591884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.591898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.591960 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:36.593161 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.593651 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.595178 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.595369 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.595449 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.602924 (+ 7475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:36.602926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:36.602926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:36.602933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:36.604598 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.604993 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.604995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.604995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.605059 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.605323 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.605324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.605324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.605485 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:36.605563 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.605807 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.605908 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:21:36.606607 2473 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45061
I20260927 03:21:36.606716 2473 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:36.606935 2473 heartbeater.cc:511] Master 127.17.52.190:45061 requested a full tablet report, sending...
I20260927 03:21:36.607304 2148 ts_manager.cc:194] Registered new tserver with Master: 01eec0e02aff44528de55fae4683661d (127.17.52.130:36141)
I20260927 03:21:36.607671 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:46461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:21:36.617684 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:36.697389 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:36.697629 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:36.697666 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:36.698787 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:36.698863 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:36.698906 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:36.700299 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:36.700560 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:36.700805 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:36.705621 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.705660 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:36.705636 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:36.705742 2478 server_base.cc:1034] running on GCE node
I20260927 03:21:36.706082 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:36.706328 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:36.707474 2478 hybrid_clock.cc:648] HybridClock initialized: now 1790479296707451 us; error 33 us; skew 500 ppm
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:36.710064 2478 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:21:36.710445 2478 webserver.cc:460] Webserver started at http://127.17.52.131:44117/ using document root <none> and password file <none>
I20260927 03:21:36.710623 2478 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:36.710661 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:36.710754 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:36.711808 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "a564c1bd38b54ea899dcec4f23bbd316"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "28b5cdc83a7693aec3289960a5e3cf82"
server_key_iv: "1f5ec6b0dafc3dedacc18c5d4be42fa8"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.712112 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "a564c1bd38b54ea899dcec4f23bbd316"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "28b5cdc83a7693aec3289960a5e3cf82"
server_key_iv: "1f5ec6b0dafc3dedacc18c5d4be42fa8"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.713404 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:21:36.714116 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.714285 2478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:36.714383 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "a564c1bd38b54ea899dcec4f23bbd316"
format_stamp: "Formatted at 2026-09-27 03:21:36 on dist-test-slave-7j43"
server_key: "28b5cdc83a7693aec3289960a5e3cf82"
server_key_iv: "1f5ec6b0dafc3dedacc18c5d4be42fa8"
server_key_version: "encryptionkey@0"
I20260927 03:21:36.714462 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:36.730979 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:36.733382 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:36.733539 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:36.733796 2478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:36.734087 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:36.734140 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.734180 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:36.734202 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:36.739951 2478 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:38613
I20260927 03:21:36.740000 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:38613 every 8 connection(s)
I20260927 03:21:36.740293 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:36.742188 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2478
I20260927 03:21:36.742290 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:36.742513 17618 external_mini_cluster.cc:1428] Setting key 029fe7e2105cb984e902b34a8fc9e5a8
Sep 27 03:21:36 dist-test-slave-7j43 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479296, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:36.749310 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.740969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:39017 (local address 127.17.52.190:45061)
0927 03:21:36.741039 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.741041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.741247 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:21:36.741752 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.741754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.741766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.741820 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:36.742923 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.743428 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.744447 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.744730 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.744815 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.746745 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:36.746747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:36.746747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:36.746755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:36.747927 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.748317 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.748318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.748319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.748376 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.748659 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.748660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.748660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.748779 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:36.748870 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.749139 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.749219 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:21:36.749507 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.740953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45061 (local address 127.17.52.131:39017)
0927 03:21:36.741125 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.741157 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.741645 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.741890 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.741893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.742526 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:36.742824 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.742831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.743522 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.743523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.744349 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.744351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.744824 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.744825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.744961 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.745368 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:36.745376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:36.746648 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:36.748017 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.748023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.748025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.748238 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.748450 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:36.748451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:36.748452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:36.748557 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:36.748862 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:36.748868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:36.749002 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.749104 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.749254 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260927 03:21:36.749922 2607 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45061
I20260927 03:21:36.749998 2607 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:36.750128 2607 heartbeater.cc:511] Master 127.17.52.190:45061 requested a full tablet report, sending...
I20260927 03:21:36.750411 2148 ts_manager.cc:194] Registered new tserver with Master: a564c1bd38b54ea899dcec4f23bbd316 (127.17.52.131:38613)
I20260927 03:21:36.750782 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:39017
I20260927 03:21:36.754212 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:36.761247 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.755622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.17.52.190:45061)
0927 03:21:36.755677 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.755679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.755781 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:21:36.755914 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.755915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.755925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.755984 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:36.756492 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.757024 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.757717 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.757890 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.757938 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.759080 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:36.759081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:36.759081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:36.759085 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:36.760232 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.760532 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.760533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.760533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.760634 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:36.760827 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:36.760828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:36.760829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:36.760964 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:36.761036 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.761133 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.761190 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:21:36.762816 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:36.764284 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:36.770565 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:36141 (local address 127.0.0.1:35510)
0927 03:21:36.766640 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.766682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.766762 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.767123 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.767124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.767127 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:36.767173 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.767175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768214 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.768215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.769519 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.769522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.770346 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.770347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.770409 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.770420 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.770476 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.770513 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":38,"thread_start_us":65,"threads_started":1}
I20260927 03:21:36.770608 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.17.52.130:36141)
0927 03:21:36.766896 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.766901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.766910 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:36.766950 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.766952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.766968 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.767040 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:36.767266 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768110 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.769619 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.770265 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.770340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.770395 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.770488 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.770526 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":71,"threads_started":1}
I20260927 03:21:36.771406 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 (DEFAULT_TABLE table=test-table [id=aa32643cb8814ce1a8dbfccaf928fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:36.771739 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38613 (local address 127.0.0.1:36852)
0927 03:21:36.766875 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.766884 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.766935 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.767157 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.767157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.767159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:36.767199 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.767200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768493 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.768494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.769810 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.769812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.771477 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.771478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.771542 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.771546 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.771625 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.771668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":112,"thread_start_us":83,"threads_started":1}
I20260927 03:21:36.771823 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a68c1fac6c4fe7820979b9de3a7240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:36.771863 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:43243 (local address 127.0.0.1:33720)
0927 03:21:36.766746 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.766757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.766817 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.767097 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.767101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.767103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:36.767167 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.767169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768533 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.768534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.769810 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.769812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.771568 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.771569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.771663 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.771668 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.771731 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.771780 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":51,"thread_start_us":76,"threads_started":1}
I20260927 03:21:36.772070 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.17.52.129:43243)
0927 03:21:36.766793 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.766797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.766809 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:21:36.766897 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.766900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.766918 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.766997 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:36.767259 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768408 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.769900 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.770589 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.770658 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.770715 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.771943 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.771999 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":74,"threads_started":1}
I20260927 03:21:36.772212 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.766621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.17.52.131:38613)
0927 03:21:36.766786 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.766791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.766933 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:21:36.767001 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.767003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.767018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.767086 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:36.767274 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.768392 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.770760 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.771404 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.771479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.771528 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.771709 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.771752 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260927 03:21:36.772851 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 (DEFAULT_TABLE table=test-table [id=aa32643cb8814ce1a8dbfccaf928fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:36.773159 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a68c1fac6c4fe7820979b9de3a7240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:36.774991 2621 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Bootstrap starting.
I20260927 03:21:36.775681 2621 tablet_bootstrap.cc:654] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:36.775578 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 (DEFAULT_TABLE table=test-table [id=aa32643cb8814ce1a8dbfccaf928fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:36.775904 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a68c1fac6c4fe7820979b9de3a7240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:36.776400 2621 log.cc:824] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:36.777190 2621 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: No bootstrap required, opened a new log
I20260927 03:21:36.777266 2621 ts_tablet_manager.cc:1397] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:36.779343 2624 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: Bootstrap starting.
I20260927 03:21:36.779536 2622 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Bootstrap starting.
I20260927 03:21:36.779464 2621 raft_consensus.cc:348] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.779641 2621 raft_consensus.cc:374] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:36.779680 2621 raft_consensus.cc:729] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01eec0e02aff44528de55fae4683661d, State: Initialized, Role: FOLLOWER
I20260927 03:21:36.779809 2621 consensus_queue.cc:260] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.780066 2624 tablet_bootstrap.cc:654] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:36.780092 2621 ts_tablet_manager.cc:1428] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:21:36.780164 2622 tablet_bootstrap.cc:654] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:36.780224 2473 heartbeater.cc:503] Master 127.17.52.190:45061 was elected leader, sending a full tablet report...
I20260927 03:21:36.780732 2624 log.cc:824] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:36.780763 2622 log.cc:824] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:36.781492 2624 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: No bootstrap required, opened a new log
I20260927 03:21:36.781577 2624 ts_tablet_manager.cc:1397] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:36.782949 2624 raft_consensus.cc:348] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.783113 2624 raft_consensus.cc:374] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:36.783159 2624 raft_consensus.cc:729] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597dda84b38c4115acccca5f858cd37c, State: Initialized, Role: FOLLOWER
I20260927 03:21:36.783277 2624 consensus_queue.cc:260] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.783545 2624 ts_tablet_manager.cc:1428] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:36.783572 2339 heartbeater.cc:503] Master 127.17.52.190:45061 was elected leader, sending a full tablet report...
I20260927 03:21:36.783957 2622 tablet_bootstrap.cc:492] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: No bootstrap required, opened a new log
I20260927 03:21:36.784025 2622 ts_tablet_manager.cc:1397] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 03:21:36.785565 2622 raft_consensus.cc:348] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.785688 2622 raft_consensus.cc:374] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:36.785709 2622 raft_consensus.cc:729] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a564c1bd38b54ea899dcec4f23bbd316, State: Initialized, Role: FOLLOWER
I20260927 03:21:36.785781 2622 consensus_queue.cc:260] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.786001 2622 ts_tablet_manager.cc:1428] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:36.786033 2607 heartbeater.cc:503] Master 127.17.52.190:45061 was elected leader, sending a full tablet report...
W20260927 03:21:36.840888 2474 tablet.cc:2369] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:21:36.889531 2625 raft_consensus.cc:482] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:36.889658 2625 raft_consensus.cc:504] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.890040 2625 leader_election.cc:288] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597dda84b38c4115acccca5f858cd37c (127.17.52.129:43243), a564c1bd38b54ea899dcec4f23bbd316 (127.17.52.131:38613)
I20260927 03:21:36.892951 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.890223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:43243 (local address 127.17.52.130:44735)
0927 03:21:36.890285 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.890299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.890363 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.890550 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.890552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.890554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:36.890629 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.890630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.891282 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.891283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.892076 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.892078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.892677 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.892764 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.892766 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.892837 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.892879 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32}
I20260927 03:21:36.892951 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.890331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:44735 (local address 127.17.52.129:43243)
0927 03:21:36.890399 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.890401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.890411 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:21:36.890442 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.890443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.890457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.890535 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:36.890718 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.891191 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892181 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.892590 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892681 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.892691 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.892844 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.892881 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:21:36.893064 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.890455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:40191 (local address 127.17.52.131:38613)
0927 03:21:36.890497 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.890499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.890582 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:21:36.890614 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.890615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.890624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.890691 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:36.890855 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.891309 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892273 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.892722 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.892811 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.892964 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.893001 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:21:36.893077 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.890352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:38613 (local address 127.17.52.130:40191)
0927 03:21:36.890501 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.890511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:36.890558 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:36.890694 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:36.890694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:36.890696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:36.890768 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.890770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.891397 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.891398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.892177 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:36.892179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.892805 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.892806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:36.892880 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.892883 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:36.892958 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:21:36.892996 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260927 03:21:36.893316 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a68c1fac6c4fe7820979b9de3a7240" candidate_uuid: "01eec0e02aff44528de55fae4683661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597dda84b38c4115acccca5f858cd37c" is_pre_election: true
I20260927 03:21:36.893469 2293 raft_consensus.cc:2454] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01eec0e02aff44528de55fae4683661d in term 0.
I20260927 03:21:36.893564 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a68c1fac6c4fe7820979b9de3a7240" candidate_uuid: "01eec0e02aff44528de55fae4683661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a564c1bd38b54ea899dcec4f23bbd316" is_pre_election: true
I20260927 03:21:36.893743 2363 leader_election.cc:302] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01eec0e02aff44528de55fae4683661d, 597dda84b38c4115acccca5f858cd37c; no voters:
I20260927 03:21:36.893781 2561 raft_consensus.cc:2454] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01eec0e02aff44528de55fae4683661d in term 0.
I20260927 03:21:36.893898 2625 raft_consensus.cc:2781] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:36.893963 2625 raft_consensus.cc:482] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:36.893981 2625 raft_consensus.cc:3037] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:36.894737 2625 raft_consensus.cc:504] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.894850 2625 leader_election.cc:288] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [CANDIDATE]: Term 1 election: Requested vote from peers 597dda84b38c4115acccca5f858cd37c (127.17.52.129:43243), a564c1bd38b54ea899dcec4f23bbd316 (127.17.52.131:38613)
I20260927 03:21:36.895021 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a68c1fac6c4fe7820979b9de3a7240" candidate_uuid: "01eec0e02aff44528de55fae4683661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597dda84b38c4115acccca5f858cd37c"
I20260927 03:21:36.895047 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a68c1fac6c4fe7820979b9de3a7240" candidate_uuid: "01eec0e02aff44528de55fae4683661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a564c1bd38b54ea899dcec4f23bbd316"
I20260927 03:21:36.895113 2561 raft_consensus.cc:3037] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:36.895118 2293 raft_consensus.cc:3037] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:36.895851 2293 raft_consensus.cc:2454] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01eec0e02aff44528de55fae4683661d in term 1.
I20260927 03:21:36.895893 2561 raft_consensus.cc:2454] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01eec0e02aff44528de55fae4683661d in term 1.
I20260927 03:21:36.896070 2363 leader_election.cc:302] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01eec0e02aff44528de55fae4683661d, a564c1bd38b54ea899dcec4f23bbd316; no voters:
I20260927 03:21:36.896166 2625 raft_consensus.cc:2781] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:36.896361 2625 raft_consensus.cc:686] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 1 LEADER]: Becoming Leader. State: Replica: 01eec0e02aff44528de55fae4683661d, State: Running, Role: LEADER
I20260927 03:21:36.896466 2625 consensus_queue.cc:237] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } }
I20260927 03:21:36.897367 2148 catalog_manager.cc:5495] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01eec0e02aff44528de55fae4683661d (127.17.52.130). New cstate: current_term: 1 leader_uuid: "01eec0e02aff44528de55fae4683661d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01eec0e02aff44528de55fae4683661d" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 } health_report { overall_health: UNKNOWN } } }
I20260927 03:21:36.914294 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.911804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.17.52.130:36141)
0927 03:21:36.911865 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:36.911867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:36.911888 (+ 21us) server_negotiation.cc:407] Connection header received
0927 03:21:36.911921 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:36.911921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:36.911946 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:36.912019 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:36.912169 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.912714 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.913476 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:36.913680 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:36.913763 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:36.913936 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:36.914045 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:36.914133 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:36.914162 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:21:36.914181 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:21:36.916095 2561 raft_consensus.cc:1261] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 1 FOLLOWER]: Refusing update from remote peer 01eec0e02aff44528de55fae4683661d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:36.916093 2293 raft_consensus.cc:1261] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 1 FOLLOWER]: Refusing update from remote peer 01eec0e02aff44528de55fae4683661d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:36.916478 2625 consensus_queue.cc:1035] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a564c1bd38b54ea899dcec4f23bbd316" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 38613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:36.916625 2631 consensus_queue.cc:1035] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "597dda84b38c4115acccca5f858cd37c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:36.918782 2634 mvcc.cc:205] Tried to move back new op lower bound from 7333803200165355520 to 7333803200088535040. Current Snapshot: MvccSnapshot[applied={T|T < 7333803200165355520}]
I20260927 03:21:36.919991 2636 mvcc.cc:205] Tried to move back new op lower bound from 7333803200165355520 to 7333803200088535040. Current Snapshot: MvccSnapshot[applied={T|T < 7333803200165355520 or (T in {7333803200165355520})}]
I20260927 03:21:36.921273 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:36.921384 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:36.922163 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9404160ca7d049b6a76f70c79428d7ae: Sending DeleteTablet for 3 replicas of tablet 38a68c1fac6c4fe7820979b9de3a7240
I20260927 03:21:36.922474 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:36 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:35510
I20260927 03:21:36.922520 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:36 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:36852
I20260927 03:21:36.922528 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a68c1fac6c4fe7820979b9de3a7240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:36 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:33720
I20260927 03:21:36.922786 2645 tablet_replica.cc:330] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c: stopping tablet replica
I20260927 03:21:36.922895 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2210
I20260927 03:21:36.922988 2645 raft_consensus.cc:2229] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:36.923096 2645 raft_consensus.cc:2258] T 38a68c1fac6c4fe7820979b9de3a7240 P 597dda84b38c4115acccca5f858cd37c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:36.923240 2643 tablet_replica.cc:330] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: stopping tablet replica
I20260927 03:21:36.923319 2643 raft_consensus.cc:2229] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:21:36.923432 2643 raft_consensus.cc:2258] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:36.923563 2644 tablet_replica.cc:330] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: stopping tablet replica
I20260927 03:21:36.923655 2644 raft_consensus.cc:2229] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:36.923744 2644 raft_consensus.cc:2258] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:36.924225 2643 ts_tablet_manager.cc:1905] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:36.926220 2643 ts_tablet_manager.cc:1918] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:36.926283 2643 log.cc:1196] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/38a68c1fac6c4fe7820979b9de3a7240
I20260927 03:21:36.926537 2643 ts_tablet_manager.cc:1939] T 38a68c1fac6c4fe7820979b9de3a7240 P 01eec0e02aff44528de55fae4683661d: Deleting consensus metadata
I20260927 03:21:36.927167 2134 catalog_manager.cc:4841] TS 01eec0e02aff44528de55fae4683661d (127.17.52.130:36141): tablet 38a68c1fac6c4fe7820979b9de3a7240 (table test-table [id=aa32643cb8814ce1a8dbfccaf928fad5]) successfully deleted
W20260927 03:21:36.928431 2135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:43243 (error 108)
I20260927 03:21:36.928922 2644 ts_tablet_manager.cc:1905] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:36.929502 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:36.929359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:43243 (local address 127.0.0.1:33726)
0927 03:21:36.929418 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:36.929461 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:43243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:21:36.929553 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2344
W20260927 03:21:36.929728 2135 catalog_manager.cc:4584] TS 597dda84b38c4115acccca5f858cd37c (127.17.52.129:43243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38a68c1fac6c4fe7820979b9de3a7240: Network error: Client connection negotiation failed: client connection to 127.17.52.129:43243: connect: Connection refused (error 111)
I20260927 03:21:36.930714 2644 ts_tablet_manager.cc:1918] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:36.930806 2644 log.cc:1196] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/38a68c1fac6c4fe7820979b9de3a7240
I20260927 03:21:36.931093 2644 ts_tablet_manager.cc:1939] T 38a68c1fac6c4fe7820979b9de3a7240 P a564c1bd38b54ea899dcec4f23bbd316: Deleting consensus metadata
I20260927 03:21:36.931530 2135 catalog_manager.cc:4841] TS a564c1bd38b54ea899dcec4f23bbd316 (127.17.52.131:38613): tablet 38a68c1fac6c4fe7820979b9de3a7240 (table test-table [id=aa32643cb8814ce1a8dbfccaf928fad5]) successfully deleted
I20260927 03:21:36.934652 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2478
I20260927 03:21:36.938968 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2116
2026-09-27T03:21:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T03:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:36Z Disabled control of system clock
I20260927 03:21:36.966130 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40071
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:40071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:37.045802 2650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:37.046025 2650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:37.046046 2650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:37.047279 2650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:37.047326 2650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:37.047339 2650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:37.047351 2650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:37.047395 2650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:37.048795 2650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:40071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:37.049059 2650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:37.049381 2650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:37.054278 2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.054250 2656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.054373 2650 server_base.cc:1034] running on GCE node
W20260927 03:21:37.054245 2658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.054658 2650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:37.054966 2650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:37.056118 2650 hybrid_clock.cc:648] HybridClock initialized: now 1790479297056106 us; error 30 us; skew 500 ppm
I20260927 03:21:37.057467 2650 webserver.cc:460] Webserver started at http://127.17.52.190:41741/ using document root <none> and password file <none>
I20260927 03:21:37.057742 2650 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:37.057792 2650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:37.057964 2650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:37.059012 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "bbdce664960448c188c6dce4b5f934ed"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "245b13301bed03482213f4bcb6171628"
server_key_iv: "2c087218e2aaf0179157aad67f6ba376"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.059373 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "bbdce664960448c188c6dce4b5f934ed"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "245b13301bed03482213f4bcb6171628"
server_key_iv: "2c087218e2aaf0179157aad67f6ba376"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.060732 2650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:37.061439 2664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.061622 2650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:37.061713 2650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "bbdce664960448c188c6dce4b5f934ed"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "245b13301bed03482213f4bcb6171628"
server_key_iv: "2c087218e2aaf0179157aad67f6ba376"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.061784 2650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:37.103477 2650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:37.103833 2650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:37.103998 2650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:37.108059 2650 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:40071
I20260927 03:21:37.108106 2716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:40071 every 8 connection(s)
I20260927 03:21:37.108475 2650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:37.109444 2717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:37.110549 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2650
I20260927 03:21:37.110667 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:37.110867 17618 external_mini_cluster.cc:1428] Setting key 0e71391a31c729620839de969c3d3c02
I20260927 03:21:37.112106 2717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: Bootstrap starting.
I20260927 03:21:37.112810 2717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:37.113130 2717 log.cc:824] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:37.113844 2717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: No bootstrap required, opened a new log
I20260927 03:21:37.115904 2717 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } }
I20260927 03:21:37.116093 2717 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:37.116135 2717 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdce664960448c188c6dce4b5f934ed, State: Initialized, Role: FOLLOWER
I20260927 03:21:37.116289 2717 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } }
I20260927 03:21:37.116382 2717 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:37.116425 2717 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:37.116472 2717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:37.116858 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.111369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.17.52.190:40071)
0927 03:21:37.111579 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.111589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.111622 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:21:37.112128 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.112137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.112218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.112365 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:37.112521 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.113822 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.114598 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.115210 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.115358 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.115995 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:37.115999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:37.115999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:37.116214 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:37.116258 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:37.116272 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:37.116350 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.116556 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.116675 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260927 03:21:37.117295 2717 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } }
I20260927 03:21:37.117396 2717 leader_election.cc:302] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdce664960448c188c6dce4b5f934ed; no voters:
I20260927 03:21:37.117553 2717 leader_election.cc:288] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:37.117607 2722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:37.117763 2722 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [term 1 LEADER]: Becoming Leader. State: Replica: bbdce664960448c188c6dce4b5f934ed, State: Running, Role: LEADER
I20260927 03:21:37.117791 2717 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:37.117892 2722 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } }
I20260927 03:21:37.118273 2723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbdce664960448c188c6dce4b5f934ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } } }
I20260927 03:21:37.118276 2724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdce664960448c188c6dce4b5f934ed. Latest consensus state: current_term: 1 leader_uuid: "bbdce664960448c188c6dce4b5f934ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdce664960448c188c6dce4b5f934ed" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40071 } } }
I20260927 03:21:37.118456 2724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:37.118412 2723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:37.118932 2731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:37.119431 2731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:37.120924 2731 catalog_manager.cc:1347] Generated new cluster ID: 55452f4c41fc4feb8df3bd22f87e6fd7
I20260927 03:21:37.120981 2731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:37.139454 2731 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:37.140173 2731 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:37.145589 2731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: Generated new TSK 0
I20260927 03:21:37.145764 2731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:37.147125 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--builtin_ntp_servers=127.17.52.148:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:37.247473 2741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:37.247673 2741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:37.247694 2741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:37.248917 2741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:37.248957 2741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:37.248983 2741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:37.250319 2741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:37.250566 2741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:37.250880 2741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:37.255741 2746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.255741 2749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.256012 2741 server_base.cc:1034] running on GCE node
W20260927 03:21:37.256047 2747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.256238 2741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:37.256498 2741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:37.257661 2741 hybrid_clock.cc:648] HybridClock initialized: now 1790479297257651 us; error 43 us; skew 500 ppm
I20260927 03:21:37.259039 2741 webserver.cc:460] Webserver started at http://127.17.52.129:46837/ using document root <none> and password file <none>
I20260927 03:21:37.259286 2741 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:37.259368 2741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:37.259505 2741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:37.260528 2741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "e8d076c4411e4ba493546b9119c0beb8"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "61d4dcaa3f3d10330648d71fd2e849e5"
server_key_iv: "0bcf67994ec13bcf011a4eb751a5ee3b"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.260957 2741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "e8d076c4411e4ba493546b9119c0beb8"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "61d4dcaa3f3d10330648d71fd2e849e5"
server_key_iv: "0bcf67994ec13bcf011a4eb751a5ee3b"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.262256 2741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:37.263048 2755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.263226 2741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:37.263339 2741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "e8d076c4411e4ba493546b9119c0beb8"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "61d4dcaa3f3d10330648d71fd2e849e5"
server_key_iv: "0bcf67994ec13bcf011a4eb751a5ee3b"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.263407 2741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:37.273193 2741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:37.273514 2741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:37.273661 2741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:37.273880 2741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:37.274200 2741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:37.274258 2741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.274299 2741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:37.274331 2741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.279850 2741 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46715
I20260927 03:21:37.279932 2868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46715 every 8 connection(s)
I20260927 03:21:37.280261 2741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:37.282820 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2741
I20260927 03:21:37.282917 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:37.283133 17618 external_mini_cluster.cc:1428] Setting key 4bfef68015173a192c62fd35f8c263cf
I20260927 03:21:37.284107 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--builtin_ntp_servers=127.17.52.148:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:21:37.286114 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.281086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:54839 (local address 127.17.52.190:40071)
0927 03:21:37.281161 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.281163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.281409 (+ 246us) server_negotiation.cc:407] Connection header received
0927 03:21:37.281964 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.281965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.281976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.282038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:37.282765 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.283448 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.284716 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.284933 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.285020 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.285863 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:37.285865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:37.285865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:37.285870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:37.285884 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:37.285888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:37.285937 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.286016 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.286039 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:21:37.286367 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.281079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40071 (local address 127.17.52.129:54839)
0927 03:21:37.281281 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.281315 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.281847 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.282115 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.282118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.282126 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:37.282662 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.282668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.283544 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.283545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.284556 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.284628 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.285033 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.285034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.285167 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.285718 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:37.285718 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:37.285746 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:37.285768 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:37.285936 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:37.285942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:37.285945 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.286012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.286112 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260927 03:21:37.286828 2869 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40071
I20260927 03:21:37.286929 2869 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:37.287102 2869 heartbeater.cc:511] Master 127.17.52.190:40071 requested a full tablet report, sending...
I20260927 03:21:37.287611 2681 ts_manager.cc:194] Registered new tserver with Master: e8d076c4411e4ba493546b9119c0beb8 (127.17.52.129:46715)
I20260927 03:21:37.288641 2681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.129:54839
W20260927 03:21:37.364843 2872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:37.365039 2872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:37.365059 2872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:37.366225 2872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:37.366262 2872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:37.366287 2872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:37.367672 2872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:37.367857 2872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:37.368080 2872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:37.372864 2880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.372864 2878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.372893 2877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.373088 2872 server_base.cc:1034] running on GCE node
I20260927 03:21:37.373286 2872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:37.373536 2872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:37.374739 2872 hybrid_clock.cc:648] HybridClock initialized: now 1790479297374728 us; error 38 us; skew 500 ppm
I20260927 03:21:37.375984 2872 webserver.cc:460] Webserver started at http://127.17.52.130:37025/ using document root <none> and password file <none>
I20260927 03:21:37.376233 2872 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:37.376305 2872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:37.376457 2872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:37.377529 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "09311432eff84fb298b3f93a0f1af95e"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "7b10a33e86be133bedb608e6412835cc"
server_key_iv: "d1a86f13b2e41ffcfc8078e57618698d"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.377897 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "09311432eff84fb298b3f93a0f1af95e"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "7b10a33e86be133bedb608e6412835cc"
server_key_iv: "d1a86f13b2e41ffcfc8078e57618698d"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.379197 2872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:21:37.380016 2886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.380227 2872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:37.380339 2872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "09311432eff84fb298b3f93a0f1af95e"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "7b10a33e86be133bedb608e6412835cc"
server_key_iv: "d1a86f13b2e41ffcfc8078e57618698d"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.380432 2872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:37.400655 2872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:37.400985 2872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:37.401149 2872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:37.401404 2872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:37.401770 2872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:37.401830 2872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.401877 2872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:37.401901 2872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.407624 2872 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:38527
I20260927 03:21:37.407666 2999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:38527 every 8 connection(s)
I20260927 03:21:37.408007 2872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:37.409476 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 2872
I20260927 03:21:37.409590 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:37.409796 17618 external_mini_cluster.cc:1428] Setting key 513a8914ac943911c79c22cc6b021fe6
I20260927 03:21:37.410816 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--builtin_ntp_servers=127.17.52.148:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:21:37.413791 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.408698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:53163 (local address 127.17.52.190:40071)
0927 03:21:37.408759 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.408761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.409009 (+ 248us) server_negotiation.cc:407] Connection header received
0927 03:21:37.409589 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.409590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.409603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.409648 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:37.410442 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.411071 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.412225 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.412416 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.412474 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.413475 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:37.413477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:37.413477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:37.413482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:37.413498 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:37.413502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:37.413551 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.413642 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.413670 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:21:37.413978 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.408734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40071 (local address 127.17.52.130:53163)
0927 03:21:37.408889 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.408924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.409463 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.409734 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.409738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.409746 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:37.410345 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.410351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.411184 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.411185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.412121 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.412124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.412504 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.412505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.412690 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.413327 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:37.413328 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:37.413357 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:37.413380 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:37.413551 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:37.413555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:37.413558 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.413646 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.413747 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260927 03:21:37.414495 3000 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40071
I20260927 03:21:37.414594 3000 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:37.414803 3000 heartbeater.cc:511] Master 127.17.52.190:40071 requested a full tablet report, sending...
I20260927 03:21:37.415216 2681 ts_manager.cc:194] Registered new tserver with Master: 09311432eff84fb298b3f93a0f1af95e (127.17.52.130:38527)
I20260927 03:21:37.415802 2681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.130:53163
W20260927 03:21:37.492611 3003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:37.492805 3003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:37.492827 3003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:37.493881 3003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:37.493913 3003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:37.493937 3003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:37.495177 3003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:37.495415 3003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:37.495669 3003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:37.500588 3011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.500661 3009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:37.500564 3008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:37.500831 3003 server_base.cc:1034] running on GCE node
I20260927 03:21:37.501071 3003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:37.501346 3003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:37.502517 3003 hybrid_clock.cc:648] HybridClock initialized: now 1790479297502494 us; error 33 us; skew 500 ppm
I20260927 03:21:37.503842 3003 webserver.cc:460] Webserver started at http://127.17.52.131:42465/ using document root <none> and password file <none>
I20260927 03:21:37.504103 3003 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:37.504197 3003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:37.504357 3003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:37.505491 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "4427c6c12b114fc785d0ec64da493e38"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "c0e30ea6c5d715d2c00d467e554a6c43"
server_key_iv: "bf66084330f62865d463272f5c0db389"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.505887 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "4427c6c12b114fc785d0ec64da493e38"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "c0e30ea6c5d715d2c00d467e554a6c43"
server_key_iv: "bf66084330f62865d463272f5c0db389"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.507167 3003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:21:37.507858 3017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.508029 3003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:37.508133 3003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "4427c6c12b114fc785d0ec64da493e38"
format_stamp: "Formatted at 2026-09-27 03:21:37 on dist-test-slave-7j43"
server_key: "c0e30ea6c5d715d2c00d467e554a6c43"
server_key_iv: "bf66084330f62865d463272f5c0db389"
server_key_version: "encryptionkey@0"
I20260927 03:21:37.508237 3003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:37.532768 3003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:37.533102 3003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:37.533280 3003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:37.533514 3003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:37.533854 3003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:37.533910 3003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.533955 3003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:37.533985 3003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:37.539696 3003 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42633
I20260927 03:21:37.539746 3130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42633 every 8 connection(s)
I20260927 03:21:37.540087 3003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:37.545222 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.540802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:38963 (local address 127.17.52.190:40071)
0927 03:21:37.540895 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.540897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.541150 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:21:37.541603 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.541604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.541615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.541663 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:37.542364 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.542846 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.543874 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.544049 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.544098 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.544999 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:37.545000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:37.545000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:37.545004 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:37.545017 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:37.545020 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:37.545063 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.545144 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.545164 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 03:21:37.545469 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.540787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40071 (local address 127.17.52.131:38963)
0927 03:21:37.541015 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.541052 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.541517 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.541764 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.541768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.541776 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:37.542280 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.542284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.542956 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.542957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.543783 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.543786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.544125 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.544126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.544241 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.544864 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:37.544864 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:37.544892 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:37.544914 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:37.545078 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:37.545081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:37.545084 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.545135 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.545215 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":128,"threads_started":1}
I20260927 03:21:37.546012 3131 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40071
I20260927 03:21:37.546103 3131 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:37.546175 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3003
I20260927 03:21:37.546278 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:37.546330 3131 heartbeater.cc:511] Master 127.17.52.190:40071 requested a full tablet report, sending...
I20260927 03:21:37.546485 17618 external_mini_cluster.cc:1428] Setting key eac9248ceffd3ff8ea276c547f604669
I20260927 03:21:37.546716 2681 ts_manager.cc:194] Registered new tserver with Master: 4427c6c12b114fc785d0ec64da493e38 (127.17.52.131:42633)
I20260927 03:21:37.547161 2681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.131:38963
I20260927 03:21:37.547394 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:37.551733 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.548807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.17.52.190:40071)
0927 03:21:37.548851 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.548854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.548920 (+ 66us) server_negotiation.cc:407] Connection header received
0927 03:21:37.548995 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.548996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.549006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.549060 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:37.549245 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.549712 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.550476 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.550692 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.550787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.551459 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:37.551461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:37.551461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:37.551466 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:37.551481 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:37.551489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:37.551549 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.551617 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.551639 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:21:37.553530 2681 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:37.555081 2681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:37.562300 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.17.52.131:42633)
0927 03:21:37.557802 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.557807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.557892 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:21:37.557943 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.557946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.557961 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.558048 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:37.558222 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.559349 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.561127 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.561847 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.561901 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.561964 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.562150 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.562207 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":88,"threads_started":1}
I20260927 03:21:37.562582 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42633 (local address 127.0.0.1:58950)
0927 03:21:37.557800 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.557809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.557861 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.558055 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.558058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.558061 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:37.558145 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.558148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.559520 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.559521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.561034 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.561037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.561928 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.561929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.562009 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.562022 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.562446 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.562514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":79,"thread_start_us":96,"threads_started":1}
I20260927 03:21:37.563052 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:38527 (local address 127.0.0.1:40046)
0927 03:21:37.558130 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.558140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.558200 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.559592 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.559593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.559617 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:37.559662 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.559664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.560694 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.560695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.561596 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.561599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.562819 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.562820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.562892 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.562896 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.562951 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.562990 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":67,"threads_started":1}
I20260927 03:21:37.563231 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.17.52.129:46715)
0927 03:21:37.557889 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.557892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.557903 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:21:37.557942 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.557944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.557961 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.558038 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:37.558369 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.559464 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.561996 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.562793 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.562873 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.562961 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.563014 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.563050 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":102,"threads_started":1}
I20260927 03:21:37.563278 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.17.52.130:38527)
0927 03:21:37.557773 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.557777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.559441 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 03:21:37.559489 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.559491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.559509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.559578 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:37.559732 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.560541 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.561683 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.562610 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.562681 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.562782 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.563163 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.563202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20260927 03:21:37.563360 3065 tablet_service.cc:1467] Processing CreateTablet for tablet a2deac67c8954f788898287bdf314fec (DEFAULT_TABLE table=test-table [id=c0f4ce44d33b4363a38eab3cd48401d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:37.563570 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.557262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46715 (local address 127.0.0.1:56396)
0927 03:21:37.557584 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.557627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.557697 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.558256 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.558257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.558258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:37.558295 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.558296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.559538 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.559538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.561034 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.561037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.562876 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.562877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.562938 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.562942 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.562998 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.563040 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":61,"thread_start_us":71,"threads_started":1}
I20260927 03:21:37.563747 3065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2deac67c8954f788898287bdf314fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:37.563954 2934 tablet_service.cc:1467] Processing CreateTablet for tablet a2deac67c8954f788898287bdf314fec (DEFAULT_TABLE table=test-table [id=c0f4ce44d33b4363a38eab3cd48401d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:37.564265 2934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2deac67c8954f788898287bdf314fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:37.566677 3145 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Bootstrap starting.
I20260927 03:21:37.567184 3146 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Bootstrap starting.
I20260927 03:21:37.567292 3145 tablet_bootstrap.cc:654] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:37.567603 3145 log.cc:824] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:37.567948 3146 tablet_bootstrap.cc:654] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:37.567927 2803 tablet_service.cc:1467] Processing CreateTablet for tablet a2deac67c8954f788898287bdf314fec (DEFAULT_TABLE table=test-table [id=c0f4ce44d33b4363a38eab3cd48401d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:37.568234 2803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2deac67c8954f788898287bdf314fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:37.568282 3146 log.cc:824] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:37.568430 3145 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: No bootstrap required, opened a new log
I20260927 03:21:37.568500 3145 ts_tablet_manager.cc:1397] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:37.570472 3145 raft_consensus.cc:348] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.570642 3145 raft_consensus.cc:374] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:37.570680 3145 raft_consensus.cc:729] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4427c6c12b114fc785d0ec64da493e38, State: Initialized, Role: FOLLOWER
I20260927 03:21:37.570701 3149 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: Bootstrap starting.
I20260927 03:21:37.570808 3145 consensus_queue.cc:260] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.571084 3145 ts_tablet_manager.cc:1428] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 03:21:37.571125 3131 heartbeater.cc:503] Master 127.17.52.190:40071 was elected leader, sending a full tablet report...
I20260927 03:21:37.571631 3146 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: No bootstrap required, opened a new log
I20260927 03:21:37.571722 3146 ts_tablet_manager.cc:1397] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:21:37.573691 3146 raft_consensus.cc:348] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.573843 3146 raft_consensus.cc:374] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:37.573889 3146 raft_consensus.cc:729] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09311432eff84fb298b3f93a0f1af95e, State: Initialized, Role: FOLLOWER
I20260927 03:21:37.574000 3146 consensus_queue.cc:260] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.574247 3146 ts_tablet_manager.cc:1428] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:21:37.571343 3149 tablet_bootstrap.cc:654] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:37.574314 3000 heartbeater.cc:503] Master 127.17.52.190:40071 was elected leader, sending a full tablet report...
I20260927 03:21:37.574609 3149 log.cc:824] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:37.575331 3149 tablet_bootstrap.cc:492] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: No bootstrap required, opened a new log
I20260927 03:21:37.575385 3149 ts_tablet_manager.cc:1397] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:21:37.577206 3149 raft_consensus.cc:348] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.577327 3149 raft_consensus.cc:374] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:37.577347 3149 raft_consensus.cc:729] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d076c4411e4ba493546b9119c0beb8, State: Initialized, Role: FOLLOWER
I20260927 03:21:37.577466 3149 consensus_queue.cc:260] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.577718 3149 ts_tablet_manager.cc:1428] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:21:37.577805 2869 heartbeater.cc:503] Master 127.17.52.190:40071 was elected leader, sending a full tablet report...
I20260927 03:21:37.645753 3151 raft_consensus.cc:482] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:37.645900 3151 raft_consensus.cc:504] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.646373 3151 leader_election.cc:288] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4427c6c12b114fc785d0ec64da493e38 (127.17.52.131:42633), e8d076c4411e4ba493546b9119c0beb8 (127.17.52.129:46715)
I20260927 03:21:37.650381 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.646657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:38919 (local address 127.17.52.131:42633)
0927 03:21:37.646713 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.646715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.646723 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:21:37.646783 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.646784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.646796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.646884 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:37.647163 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.647918 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.649454 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.650008 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.650091 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.650100 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.650271 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.650313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:21:37.650381 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.646655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:54139 (local address 127.17.52.129:46715)
0927 03:21:37.646707 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.646709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.646793 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:21:37.646825 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.646826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.646838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.646902 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:37.647146 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.647918 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.649454 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.650026 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.650091 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.650105 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.650271 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.650313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:21:37.650382 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.646558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42633 (local address 127.17.52.130:38919)
0927 03:21:37.646618 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.646630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.646673 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.646913 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.646913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.646915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:37.647026 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.647028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.648005 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.648006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.649340 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.649343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.650100 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.650101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.650182 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.650186 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.650243 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.650291 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":94}
I20260927 03:21:37.650389 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.646572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46715 (local address 127.17.52.130:54139)
0927 03:21:37.646724 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.646732 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:37.646779 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:37.646913 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:37.646913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:37.646915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:37.647026 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.647028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.648005 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.648006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.649354 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:37.649356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.650101 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.650102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:37.650182 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.650186 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:37.650243 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:21:37.650289 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260927 03:21:37.650743 3085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2deac67c8954f788898287bdf314fec" candidate_uuid: "09311432eff84fb298b3f93a0f1af95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4427c6c12b114fc785d0ec64da493e38" is_pre_election: true
I20260927 03:21:37.650910 3085 raft_consensus.cc:2454] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09311432eff84fb298b3f93a0f1af95e in term 0.
I20260927 03:21:37.650934 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2deac67c8954f788898287bdf314fec" candidate_uuid: "09311432eff84fb298b3f93a0f1af95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d076c4411e4ba493546b9119c0beb8" is_pre_election: true
I20260927 03:21:37.651058 2823 raft_consensus.cc:2454] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09311432eff84fb298b3f93a0f1af95e in term 0.
I20260927 03:21:37.651297 2887 leader_election.cc:302] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09311432eff84fb298b3f93a0f1af95e, 4427c6c12b114fc785d0ec64da493e38; no voters:
I20260927 03:21:37.651444 3151 raft_consensus.cc:2781] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:37.651499 3151 raft_consensus.cc:482] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:37.651520 3151 raft_consensus.cc:3037] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:37.652240 3151 raft_consensus.cc:504] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.652498 3151 leader_election.cc:288] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [CANDIDATE]: Term 1 election: Requested vote from peers 4427c6c12b114fc785d0ec64da493e38 (127.17.52.131:42633), e8d076c4411e4ba493546b9119c0beb8 (127.17.52.129:46715)
I20260927 03:21:37.652513 3085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2deac67c8954f788898287bdf314fec" candidate_uuid: "09311432eff84fb298b3f93a0f1af95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4427c6c12b114fc785d0ec64da493e38"
I20260927 03:21:37.652513 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2deac67c8954f788898287bdf314fec" candidate_uuid: "09311432eff84fb298b3f93a0f1af95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d076c4411e4ba493546b9119c0beb8"
I20260927 03:21:37.652598 3085 raft_consensus.cc:3037] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:37.652613 2823 raft_consensus.cc:3037] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:37.653492 2823 raft_consensus.cc:2454] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09311432eff84fb298b3f93a0f1af95e in term 1.
I20260927 03:21:37.653492 3085 raft_consensus.cc:2454] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09311432eff84fb298b3f93a0f1af95e in term 1.
I20260927 03:21:37.653672 2889 leader_election.cc:302] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09311432eff84fb298b3f93a0f1af95e, e8d076c4411e4ba493546b9119c0beb8; no voters:
I20260927 03:21:37.653767 3151 raft_consensus.cc:2781] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:37.653904 3151 raft_consensus.cc:686] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 1 LEADER]: Becoming Leader. State: Replica: 09311432eff84fb298b3f93a0f1af95e, State: Running, Role: LEADER
I20260927 03:21:37.654042 3151 consensus_queue.cc:237] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } }
I20260927 03:21:37.654949 2681 catalog_manager.cc:5495] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 09311432eff84fb298b3f93a0f1af95e (127.17.52.130). New cstate: current_term: 1 leader_uuid: "09311432eff84fb298b3f93a0f1af95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09311432eff84fb298b3f93a0f1af95e" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 38527 } health_report { overall_health: HEALTHY } } }
W20260927 03:21:37.658474 3001 tablet.cc:2369] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:21:37.676146 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.673928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.17.52.130:38527)
0927 03:21:37.673987 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:37.673989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:37.674020 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:21:37.674048 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:37.674048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:37.674064 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:37.674117 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:37.674239 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.674692 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.675504 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:37.675682 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:37.675735 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:37.675864 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:37.675951 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:37.676009 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:37.676062 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:21:37.676079 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:21:37.677904 2823 raft_consensus.cc:1261] T a2deac67c8954f788898287bdf314fec P e8d076c4411e4ba493546b9119c0beb8 [term 1 FOLLOWER]: Refusing update from remote peer 09311432eff84fb298b3f93a0f1af95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:37.677904 3085 raft_consensus.cc:1261] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 1 FOLLOWER]: Refusing update from remote peer 09311432eff84fb298b3f93a0f1af95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:37.678257 3155 consensus_queue.cc:1035] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d076c4411e4ba493546b9119c0beb8" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:37.678397 3151 consensus_queue.cc:1035] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4427c6c12b114fc785d0ec64da493e38" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:37.682473 3160 mvcc.cc:205] Tried to move back new op lower bound from 7333803203286085632 to 7333803203191840768. Current Snapshot: MvccSnapshot[applied={T|T < 7333803203286085632 or (T in {7333803203286085632})}]
I20260927 03:21:37.682770 2680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:37.682874 2680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:37.683636 2680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbdce664960448c188c6dce4b5f934ed: Sending DeleteTablet for 3 replicas of tablet a2deac67c8954f788898287bdf314fec
I20260927 03:21:37.683961 3065 tablet_service.cc:1514] Processing DeleteTablet for tablet a2deac67c8954f788898287bdf314fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:37 UTC) from {username='slave'} at 127.0.0.1:58950
I20260927 03:21:37.683976 2934 tablet_service.cc:1514] Processing DeleteTablet for tablet a2deac67c8954f788898287bdf314fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:37 UTC) from {username='slave'} at 127.0.0.1:40046
I20260927 03:21:37.684201 2803 tablet_service.cc:1514] Processing DeleteTablet for tablet a2deac67c8954f788898287bdf314fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:37 UTC) from {username='slave'} at 127.0.0.1:56396
I20260927 03:21:37.684268 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2741
I20260927 03:21:37.684894 3168 tablet_replica.cc:330] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: stopping tablet replica
I20260927 03:21:37.685039 3168 raft_consensus.cc:2229] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:37.685163 3168 raft_consensus.cc:2258] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:37.685335 3167 tablet_replica.cc:330] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: stopping tablet replica
I20260927 03:21:37.685456 3167 raft_consensus.cc:2229] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:21:37.685564 3167 raft_consensus.cc:2258] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:37.685708 3168 ts_tablet_manager.cc:1905] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:37.685976 3167 ts_tablet_manager.cc:1905] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:37.687282 3168 ts_tablet_manager.cc:1918] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:37.687409 3168 log.cc:1196] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/a2deac67c8954f788898287bdf314fec
I20260927 03:21:37.687736 3168 ts_tablet_manager.cc:1939] T a2deac67c8954f788898287bdf314fec P 4427c6c12b114fc785d0ec64da493e38: Deleting consensus metadata
I20260927 03:21:37.687879 3167 ts_tablet_manager.cc:1918] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:37.687953 3167 log.cc:1196] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/a2deac67c8954f788898287bdf314fec
I20260927 03:21:37.688223 3167 ts_tablet_manager.cc:1939] T a2deac67c8954f788898287bdf314fec P 09311432eff84fb298b3f93a0f1af95e: Deleting consensus metadata
I20260927 03:21:37.688277 2665 catalog_manager.cc:4841] TS 4427c6c12b114fc785d0ec64da493e38 (127.17.52.131:42633): tablet a2deac67c8954f788898287bdf314fec (table test-table [id=c0f4ce44d33b4363a38eab3cd48401d9]) successfully deleted
I20260927 03:21:37.688650 2667 catalog_manager.cc:4841] TS 09311432eff84fb298b3f93a0f1af95e (127.17.52.130:38527): tablet a2deac67c8954f788898287bdf314fec (table test-table [id=c0f4ce44d33b4363a38eab3cd48401d9]) successfully deleted
W20260927 03:21:37.691470 2667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:46715 (error 108)
I20260927 03:21:37.691978 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:37.691837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46715 (local address 127.0.0.1:56398)
0927 03:21:37.691900 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:37.691941 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 03:21:37.692047 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2872
W20260927 03:21:37.693430 2667 catalog_manager.cc:4584] TS e8d076c4411e4ba493546b9119c0beb8 (127.17.52.129:46715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2deac67c8954f788898287bdf314fec: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46715: connect: Connection refused (error 111)
I20260927 03:21:37.697108 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3003
I20260927 03:21:37.701411 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 2650
2026-09-27T03:21:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (761 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:37 dist-test-slave-7j43 krb5kdc[3173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:37 dist-test-slave-7j43 krb5kdc[3173](info): set up 2 sockets
Sep 27 03:21:37 dist-test-slave-7j43 krb5kdc[3173](info): commencing operation
krb5kdc: starting...
W20260927 03:21:38.728901 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:38 dist-test-slave-7j43 krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:21:38.845162 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43919
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:38.924250 3186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:38.924459 3186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:38.924482 3186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:38.925653 3186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:38.925696 3186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:38.925711 3186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:38.925724 3186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:38.925736 3186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:38.927196 3186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:43919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:38.927443 3186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:38.927691 3186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:38.932642 3191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:38.932651 3192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:38.932631 3194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:38.932848 3186 server_base.cc:1034] running on GCE node
I20260927 03:21:38.933151 3186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:38.933445 3186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:38.934623 3186 hybrid_clock.cc:648] HybridClock initialized: now 1790479298934615 us; error 36 us; skew 500 ppm
Sep 27 03:21:38 dist-test-slave-7j43 krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479298, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:38.937189 3186 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:21:38.937619 3186 webserver.cc:460] Webserver started at http://127.17.52.190:44397/ using document root <none> and password file <none>
I20260927 03:21:38.937850 3186 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:38.937927 3186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:38.938083 3186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:38.939160 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a"
format_stamp: "Formatted at 2026-09-27 03:21:38 on dist-test-slave-7j43"
server_key: "5723d11cb2faa6e37474956e2cf27d5d"
server_key_iv: "e5b0e847549916202ead2f8e323e319f"
server_key_version: "encryptionkey@0"
I20260927 03:21:38.939535 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a"
format_stamp: "Formatted at 2026-09-27 03:21:38 on dist-test-slave-7j43"
server_key: "5723d11cb2faa6e37474956e2cf27d5d"
server_key_iv: "e5b0e847549916202ead2f8e323e319f"
server_key_version: "encryptionkey@0"
I20260927 03:21:38.940950 3186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:38.941676 3201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:38.941859 3186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:38.941959 3186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a"
format_stamp: "Formatted at 2026-09-27 03:21:38 on dist-test-slave-7j43"
server_key: "5723d11cb2faa6e37474956e2cf27d5d"
server_key_iv: "e5b0e847549916202ead2f8e323e319f"
server_key_version: "encryptionkey@0"
I20260927 03:21:38.942049 3186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:38.951602 3186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:38.953498 3186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:38.953662 3186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:38.957578 3186 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:43919
I20260927 03:21:38.957612 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:43919 every 8 connection(s)
I20260927 03:21:38.957957 3186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:38.958832 3254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:38.959479 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3186
I20260927 03:21:38.959600 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:38.959808 17618 external_mini_cluster.cc:1428] Setting key 7d09fb3698d08cc95e5ebf4406d85777
I20260927 03:21:38.961624 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: Bootstrap starting.
I20260927 03:21:38.962189 3254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:38.962492 3254 log.cc:824] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:38.963150 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: No bootstrap required, opened a new log
I20260927 03:21:38.965103 3254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } }
I20260927 03:21:38.965269 3254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:38.965340 3254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cb84b6c25e40dfb5c510a0ec2d1a1a, State: Initialized, Role: FOLLOWER
I20260927 03:21:38.965489 3254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } }
I20260927 03:21:38.965588 3254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:38.965631 3254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:38.965678 3254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:38.966343 3254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } }
I20260927 03:21:38.966452 3254 leader_election.cc:302] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00cb84b6c25e40dfb5c510a0ec2d1a1a; no voters:
Sep 27 03:21:38 dist-test-slave-7j43 krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:38.966614 3254 leader_election.cc:288] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:38.966672 3259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:38.966836 3259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [term 1 LEADER]: Becoming Leader. State: Replica: 00cb84b6c25e40dfb5c510a0ec2d1a1a, State: Running, Role: LEADER
I20260927 03:21:38.966871 3254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:38.966933 3259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } }
I20260927 03:21:38.967329 3260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } } }
I20260927 03:21:38.967417 3260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:38.967319 3261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00cb84b6c25e40dfb5c510a0ec2d1a1a. Latest consensus state: current_term: 1 leader_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00cb84b6c25e40dfb5c510a0ec2d1a1a" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43919 } } }
I20260927 03:21:38.967511 3261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:38.968192 3266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:38.968873 3266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:38.970458 3266 catalog_manager.cc:1347] Generated new cluster ID: 8d7196f15bca4248a8cbb5ced1726f0e
I20260927 03:21:38.970503 3266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:38.970942 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:38.960337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.17.52.190:43919)
0927 03:21:38.960520 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:38.960530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:38.960562 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:21:38.961016 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:38.961023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:38.961071 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:38.961223 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:38.961919 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:38.962927 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:38.963767 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:38.964186 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:38.964316 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:38.967138 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:38.967141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:38.967142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:38.967152 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:38.969246 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:38.969664 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:38.969667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:38.969667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:38.969772 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:38.970043 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:38.970045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:38.970045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:38.970296 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:38.970467 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:38.970632 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:21:38.970773 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260927 03:21:38.981026 3266 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:38.981688 3266 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:38.985049 3266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: Generated new TSK 0
I20260927 03:21:38.985201 3266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:21:39.015694 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:39.112903 3280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:39.113098 3280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:39.113121 3280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:39.114266 3280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:39.114306 3280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:39.114332 3280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:39.115765 3280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:39.115990 3280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:39.116235 3280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:39.122079 3285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.122083 3286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.122301 3280 server_base.cc:1034] running on GCE node
W20260927 03:21:39.122080 3288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.122676 3280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:39.122961 3280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:39.124143 3280 hybrid_clock.cc:648] HybridClock initialized: now 1790479299124127 us; error 33 us; skew 500 ppm
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:39.126746 3280 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:21:39.127120 3280 webserver.cc:460] Webserver started at http://127.17.52.129:41063/ using document root <none> and password file <none>
I20260927 03:21:39.127354 3280 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:39.127403 3280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:39.127523 3280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:39.128556 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "76384980e4134b21b4ecec653221b40b"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "dc9152cdc1a5b586b2f10a80cfb268eb"
server_key_iv: "56c4bfbdd6698842a6f292db5e74ec14"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.128989 3280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "76384980e4134b21b4ecec653221b40b"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "dc9152cdc1a5b586b2f10a80cfb268eb"
server_key_iv: "56c4bfbdd6698842a6f292db5e74ec14"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.130312 3280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:39.131091 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.131265 3280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:39.131363 3280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "76384980e4134b21b4ecec653221b40b"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "dc9152cdc1a5b586b2f10a80cfb268eb"
server_key_iv: "56c4bfbdd6698842a6f292db5e74ec14"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.131443 3280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:39.144835 3280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:39.147120 3280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:39.147282 3280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:39.147508 3280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:39.147818 3280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:39.147874 3280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.147915 3280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:39.147940 3280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.153829 3280 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:42249
I20260927 03:21:39.153882 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:42249 every 8 connection(s)
I20260927 03:21:39.154220 3280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:39.161012 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3280
I20260927 03:21:39.161150 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:39.161372 17618 external_mini_cluster.cc:1428] Setting key f6bb78e7eb8f9fac98db20aae59842c1
I20260927 03:21:39.163650 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.154970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:32927 (local address 127.17.52.190:43919)
0927 03:21:39.155037 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.155039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.155240 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:21:39.155726 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.155727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.155736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.155794 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:39.156801 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.157312 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.158279 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.158502 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.158565 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.160711 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:39.160713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:39.160713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:39.160719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:39.162250 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.162685 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.162686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.162687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.162748 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.162994 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.162995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.162996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.163111 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:39.163188 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.163463 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.163560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:21:39.163830 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.154935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43919 (local address 127.17.52.129:32927)
0927 03:21:39.155114 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.155156 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.155636 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.155880 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.155883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.156442 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:39.156713 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.156719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.157390 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.157391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.158188 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.158190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.158585 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.158586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.158686 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.159140 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:39.159148 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:39.160620 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:39.162360 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.162368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.162370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.162591 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.162817 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.162818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.162818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.162924 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.163179 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:39.163185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:39.163329 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.163441 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.163594 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260927 03:21:39.164245 3409 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43919
I20260927 03:21:39.164361 3409 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:39.164533 3409 heartbeater.cc:511] Master 127.17.52.190:43919 requested a full tablet report, sending...
I20260927 03:21:39.165104 3218 ts_manager.cc:194] Registered new tserver with Master: 76384980e4134b21b4ecec653221b40b (127.17.52.129:42249)
I20260927 03:21:39.165791 3218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:32927
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:21:39.187021 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:39.267181 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:39.267416 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:39.267454 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:39.268699 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:39.268760 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:39.268803 3414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:39.270210 3414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:39.270479 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:39.270704 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:39.275597 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.275672 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.275624 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.275820 3414 server_base.cc:1034] running on GCE node
I20260927 03:21:39.276006 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:39.276193 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:39.277338 3414 hybrid_clock.cc:648] HybridClock initialized: now 1790479299277324 us; error 27 us; skew 500 ppm
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:39.279884 3414 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:21:39.280382 3414 webserver.cc:460] Webserver started at http://127.17.52.130:46877/ using document root <none> and password file <none>
I20260927 03:21:39.280683 3414 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:39.280761 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:39.280901 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:39.281945 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "df71829de22d961903088be56fcb0f88"
server_key_iv: "79443571c563042b7717a0f0fdfeeb5f"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.282323 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "df71829de22d961903088be56fcb0f88"
server_key_iv: "79443571c563042b7717a0f0fdfeeb5f"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.283700 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:39.284518 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.284734 3414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:39.284834 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "df71829de22d961903088be56fcb0f88"
server_key_iv: "79443571c563042b7717a0f0fdfeeb5f"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.284916 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:39.308466 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:39.310613 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:39.310783 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:39.311020 3414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:39.311345 3414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:39.311400 3414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.311434 3414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:39.311460 3414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.317157 3414 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:43899
I20260927 03:21:39.317204 3542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:43899 every 8 connection(s)
I20260927 03:21:39.317516 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:39.321868 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3414
I20260927 03:21:39.322003 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:39.322234 17618 external_mini_cluster.cc:1428] Setting key f55ba8b7c807bc332922a1cf45e125a2
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:39.327924 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.318237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36063 (local address 127.17.52.190:43919)
0927 03:21:39.318297 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.318300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.318489 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:21:39.319003 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.319004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.319013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.319071 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:39.320026 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.320566 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.321566 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.321756 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.321858 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.324466 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:39.324468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:39.324469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:39.324477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:39.326264 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.326675 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.326676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.326676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.326744 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.327081 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.327082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.327083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.327275 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:39.327361 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.327720 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.327812 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:21:39.328018 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.318184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43919 (local address 127.17.52.130:36063)
0927 03:21:39.318355 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.318400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.318898 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.319161 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.319166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.319743 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:39.319936 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.319941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.320671 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.320672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.321484 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.321486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.321868 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.321869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.322028 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.322615 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:39.322625 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:39.324354 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:39.326363 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.326369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.326371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.326592 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.326877 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.326878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.326878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.326987 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.327390 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:39.327399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:39.327555 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.327668 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.327805 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260927 03:21:39.328603 3543 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43919
I20260927 03:21:39.328728 3543 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:39.328917 3543 heartbeater.cc:511] Master 127.17.52.190:43919 requested a full tablet report, sending...
I20260927 03:21:39.329305 3218 ts_manager.cc:194] Registered new tserver with Master: a9e3d37d073f469d9ab5c5ea8c1ecccb (127.17.52.130:43899)
I20260927 03:21:39.329692 3218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:36063
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:21:39.349275 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:39.429191 3548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:39.429416 3548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:39.429441 3548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:39.430603 3548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:39.430645 3548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:39.430671 3548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:39.432276 3548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:39.432544 3548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:39.432825 3548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:39.438416 3554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.438428 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.438604 3548 server_base.cc:1034] running on GCE node
W20260927 03:21:39.438673 3556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.438868 3548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:39.439110 3548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:39.440250 3548 hybrid_clock.cc:648] HybridClock initialized: now 1790479299440240 us; error 26 us; skew 500 ppm
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:39.442948 3548 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:21:39.443404 3548 webserver.cc:460] Webserver started at http://127.17.52.131:42183/ using document root <none> and password file <none>
I20260927 03:21:39.443634 3548 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:39.443706 3548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:39.443859 3548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:39.445006 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "fbf19c105cf74eb0a9f78955694b0a02"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1e3177d7481ac533f08bdd491ce357e5"
server_key_iv: "324047a21d18b8aa1c46eb65cf98b304"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.445408 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "fbf19c105cf74eb0a9f78955694b0a02"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1e3177d7481ac533f08bdd491ce357e5"
server_key_iv: "324047a21d18b8aa1c46eb65cf98b304"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.446719 3548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:39.447399 3563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.447574 3548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:39.447674 3548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "fbf19c105cf74eb0a9f78955694b0a02"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1e3177d7481ac533f08bdd491ce357e5"
server_key_iv: "324047a21d18b8aa1c46eb65cf98b304"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.447753 3548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:39.462692 3548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:39.465016 3548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:39.465179 3548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:39.465413 3548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:39.465718 3548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:39.465773 3548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.465814 3548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:39.465842 3548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.472281 3548 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:37489
I20260927 03:21:39.472318 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:37489 every 8 connection(s)
I20260927 03:21:39.472765 3548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:39.473898 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3548
I20260927 03:21:39.473974 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:39.474143 17618 external_mini_cluster.cc:1428] Setting key 341b5dfd6230ef19daa1f76336c97dcf
Sep 27 03:21:39 dist-test-slave-7j43 krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479299, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:39.482676 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.473552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:49357 (local address 127.17.52.190:43919)
0927 03:21:39.473596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.473598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.473802 (+ 204us) server_negotiation.cc:407] Connection header received
0927 03:21:39.474479 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.474481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.474494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.474546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:39.475536 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.476059 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.477195 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.477418 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.477513 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.479732 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:39.479733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:39.479734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:39.479741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:39.481233 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.481671 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.481673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.481673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.481732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.481998 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.481999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.482000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.482120 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:39.482195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.482499 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.482580 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:21:39.482877 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.473513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:43919 (local address 127.17.52.131:49357)
0927 03:21:39.473672 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.473716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.474361 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.474617 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.474621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.475217 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:39.475426 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.475431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.476164 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.476165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.477097 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.477100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.477526 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.477527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.477658 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.478161 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:39.478170 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:39.479623 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:39.481338 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.481345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.481348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.481576 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.481802 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:39.481803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:39.481804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:39.481926 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:39.482196 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:39.482204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:39.482348 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.482460 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.482628 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260927 03:21:39.483353 3677 heartbeater.cc:347] Connected to a master server at 127.17.52.190:43919
I20260927 03:21:39.483443 3677 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:39.483647 3677 heartbeater.cc:511] Master 127.17.52.190:43919 requested a full tablet report, sending...
I20260927 03:21:39.484038 3218 ts_manager.cc:194] Registered new tserver with Master: fbf19c105cf74eb0a9f78955694b0a02 (127.17.52.131:37489)
I20260927 03:21:39.484407 3218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:49357
I20260927 03:21:39.485726 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:39.492873 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.487017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.17.52.190:43919)
0927 03:21:39.487068 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.487069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.487163 (+ 94us) server_negotiation.cc:407] Connection header received
0927 03:21:39.487217 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.487217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.487225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.487270 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:39.487829 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.488286 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.489046 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.489223 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.489276 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.490429 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:39.490430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:39.490430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:39.490436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:39.491763 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.492103 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.492104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.492104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.492153 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:39.492328 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:39.492328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:39.492328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:39.492449 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:39.492526 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.492762 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.492812 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:21:39.494688 3218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:39.496299 3218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:39.502815 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:43899 (local address 127.0.0.1:39702)
0927 03:21:39.498959 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.498965 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.498996 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.499207 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.499211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.499214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:39.499300 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.499302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500761 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.500762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.501839 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.501841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502513 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.502594 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502609 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.502678 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.502736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 03:21:39.502895 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:42249 (local address 127.0.0.1:36420)
0927 03:21:39.498884 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.498891 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.498917 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.499337 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.499338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.499340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:39.499395 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.499397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500788 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.500789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.501776 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.501778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502652 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.502708 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502710 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.502760 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.502793 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":80,"threads_started":1}
I20260927 03:21:39.503063 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.17.52.131:37489)
0927 03:21:39.499114 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.499119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.499130 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:21:39.499166 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.499168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.499184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.499292 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:39.499415 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500216 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.501846 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502555 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502869 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502928 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.502954 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.502994 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":83,"threads_started":1}
I20260927 03:21:39.503147 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:37489 (local address 127.0.0.1:38230)
0927 03:21:39.498693 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.498731 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.498790 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.499293 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.499294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.499296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:39.499336 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.499338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500323 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.500324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.501748 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.501751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502660 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.502725 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502729 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.502760 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.502808 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":1,"thread_start_us":41,"threads_started":1}
I20260927 03:21:39.502935 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.17.52.129:42249)
0927 03:21:39.499175 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.499179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.499189 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:21:39.499225 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.499227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.499241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.499474 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:39.499553 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500657 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.501888 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502577 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502652 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502708 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.502807 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.502854 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260927 03:21:39.503420 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.498775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.17.52.130:43899)
0927 03:21:39.498945 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.498951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.498999 (+ 48us) server_negotiation.cc:407] Connection header received
0927 03:21:39.499062 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.499064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.499081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.499188 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:39.499496 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.500657 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.501908 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.502420 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.502515 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.502579 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.503282 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.503333 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260927 03:21:39.503898 3343 tablet_service.cc:1467] Processing CreateTablet for tablet 5a25e08af965441799d5398413c7b602 (DEFAULT_TABLE table=test-table [id=679f2fb9884e42378280df3e52d12b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:39.503998 3477 tablet_service.cc:1467] Processing CreateTablet for tablet 5a25e08af965441799d5398413c7b602 (DEFAULT_TABLE table=test-table [id=679f2fb9884e42378280df3e52d12b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:39.504181 3343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a25e08af965441799d5398413c7b602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:39.504233 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a25e08af965441799d5398413c7b602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:39.505028 3611 tablet_service.cc:1467] Processing CreateTablet for tablet 5a25e08af965441799d5398413c7b602 (DEFAULT_TABLE table=test-table [id=679f2fb9884e42378280df3e52d12b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:39.505338 3611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a25e08af965441799d5398413c7b602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:39.506695 3694 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Bootstrap starting.
I20260927 03:21:39.507403 3694 tablet_bootstrap.cc:654] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:39.507726 3695 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Bootstrap starting.
I20260927 03:21:39.507980 3694 log.cc:824] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:39.508380 3695 tablet_bootstrap.cc:654] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:39.508860 3694 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: No bootstrap required, opened a new log
I20260927 03:21:39.508919 3694 ts_tablet_manager.cc:1397] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:39.509047 3695 log.cc:824] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:39.509384 3697 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Bootstrap starting.
I20260927 03:21:39.509812 3695 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: No bootstrap required, opened a new log
I20260927 03:21:39.509892 3695 ts_tablet_manager.cc:1397] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:39.510134 3697 tablet_bootstrap.cc:654] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:39.510577 3697 log.cc:824] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:39.511087 3694 raft_consensus.cc:348] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.511279 3694 raft_consensus.cc:374] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:39.511307 3697 tablet_bootstrap.cc:492] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: No bootstrap required, opened a new log
I20260927 03:21:39.511317 3694 raft_consensus.cc:729] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e3d37d073f469d9ab5c5ea8c1ecccb, State: Initialized, Role: FOLLOWER
I20260927 03:21:39.511379 3697 ts_tablet_manager.cc:1397] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:39.511478 3694 consensus_queue.cc:260] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.511761 3694 ts_tablet_manager.cc:1428] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:39.511888 3543 heartbeater.cc:503] Master 127.17.52.190:43919 was elected leader, sending a full tablet report...
I20260927 03:21:39.511876 3695 raft_consensus.cc:348] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.512043 3695 raft_consensus.cc:374] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:39.512084 3695 raft_consensus.cc:729] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76384980e4134b21b4ecec653221b40b, State: Initialized, Role: FOLLOWER
I20260927 03:21:39.512192 3695 consensus_queue.cc:260] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.512465 3695 ts_tablet_manager.cc:1428] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:39.512476 3409 heartbeater.cc:503] Master 127.17.52.190:43919 was elected leader, sending a full tablet report...
I20260927 03:21:39.513386 3697 raft_consensus.cc:348] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.513571 3697 raft_consensus.cc:374] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:39.513608 3697 raft_consensus.cc:729] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf19c105cf74eb0a9f78955694b0a02, State: Initialized, Role: FOLLOWER
I20260927 03:21:39.513713 3697 consensus_queue.cc:260] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.513945 3697 ts_tablet_manager.cc:1428] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:39.514009 3677 heartbeater.cc:503] Master 127.17.52.190:43919 was elected leader, sending a full tablet report...
W20260927 03:21:39.568141 3544 tablet.cc:2369] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:21:39.654958 3410 tablet.cc:2369] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:21:39.671489 3702 raft_consensus.cc:482] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:39.671650 3702 raft_consensus.cc:504] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.672034 3702 leader_election.cc:288] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76384980e4134b21b4ecec653221b40b (127.17.52.129:42249), a9e3d37d073f469d9ab5c5ea8c1ecccb (127.17.52.130:43899)
I20260927 03:21:39.675011 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.672255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:42249 (local address 127.17.52.131:37081)
0927 03:21:39.672298 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.672310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.672355 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.672600 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.672602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.672604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:39.672678 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.672680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.673414 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.673415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.674190 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.674192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.674758 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.674841 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.674846 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.674904 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.674933 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:21:39.675041 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.672372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:37081 (local address 127.17.52.129:42249)
0927 03:21:39.672428 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.672430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.672439 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:39.672472 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.672473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.672483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.672571 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:39.672803 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.673337 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674287 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.674660 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674714 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.674721 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.674920 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.674963 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:21:39.675122 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.672255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:43899 (local address 127.17.52.131:51451)
0927 03:21:39.672594 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.672604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:39.672660 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:39.672794 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:39.672794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:39.672796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:39.672868 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.672870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.673496 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.673497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.674272 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:39.674274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.674853 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:39.674943 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.674947 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:39.675004 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:21:39.675038 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":151,"thread_start_us":115,"threads_started":1}
I20260927 03:21:39.675143 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.672313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:51451 (local address 127.17.52.130:43899)
0927 03:21:39.672380 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.672383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.672654 (+ 271us) server_negotiation.cc:407] Connection header received
0927 03:21:39.672701 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.672702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.672714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.672773 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:39.672936 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.673418 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674364 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.674773 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.674844 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.674852 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.675022 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.675067 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:21:39.675344 3363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a25e08af965441799d5398413c7b602" candidate_uuid: "fbf19c105cf74eb0a9f78955694b0a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76384980e4134b21b4ecec653221b40b" is_pre_election: true
I20260927 03:21:39.675464 3497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a25e08af965441799d5398413c7b602" candidate_uuid: "fbf19c105cf74eb0a9f78955694b0a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" is_pre_election: true
I20260927 03:21:39.675506 3363 raft_consensus.cc:2454] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf19c105cf74eb0a9f78955694b0a02 in term 0.
I20260927 03:21:39.675589 3497 raft_consensus.cc:2454] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf19c105cf74eb0a9f78955694b0a02 in term 0.
I20260927 03:21:39.675815 3565 leader_election.cc:302] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76384980e4134b21b4ecec653221b40b, fbf19c105cf74eb0a9f78955694b0a02; no voters:
I20260927 03:21:39.675957 3702 raft_consensus.cc:2781] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:39.676012 3702 raft_consensus.cc:482] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:39.676036 3702 raft_consensus.cc:3037] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:39.676832 3702 raft_consensus.cc:504] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.676976 3702 leader_election.cc:288] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [CANDIDATE]: Term 1 election: Requested vote from peers 76384980e4134b21b4ecec653221b40b (127.17.52.129:42249), a9e3d37d073f469d9ab5c5ea8c1ecccb (127.17.52.130:43899)
I20260927 03:21:39.677155 3497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a25e08af965441799d5398413c7b602" candidate_uuid: "fbf19c105cf74eb0a9f78955694b0a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb"
I20260927 03:21:39.677222 3497 raft_consensus.cc:3037] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:39.677192 3363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a25e08af965441799d5398413c7b602" candidate_uuid: "fbf19c105cf74eb0a9f78955694b0a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76384980e4134b21b4ecec653221b40b"
I20260927 03:21:39.677340 3363 raft_consensus.cc:3037] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:39.678025 3497 raft_consensus.cc:2454] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf19c105cf74eb0a9f78955694b0a02 in term 1.
I20260927 03:21:39.678031 3363 raft_consensus.cc:2454] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf19c105cf74eb0a9f78955694b0a02 in term 1.
I20260927 03:21:39.678190 3565 leader_election.cc:302] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76384980e4134b21b4ecec653221b40b, fbf19c105cf74eb0a9f78955694b0a02; no voters:
I20260927 03:21:39.678321 3702 raft_consensus.cc:2781] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:39.678506 3702 raft_consensus.cc:686] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 1 LEADER]: Becoming Leader. State: Replica: fbf19c105cf74eb0a9f78955694b0a02, State: Running, Role: LEADER
I20260927 03:21:39.678606 3702 consensus_queue.cc:237] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } }
I20260927 03:21:39.679543 3217 catalog_manager.cc:5495] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbf19c105cf74eb0a9f78955694b0a02 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "fbf19c105cf74eb0a9f78955694b0a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbf19c105cf74eb0a9f78955694b0a02" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 } health_report { overall_health: UNKNOWN } } }
I20260927 03:21:39.686470 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.684161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.17.52.131:37489)
0927 03:21:39.684203 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.684205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.684213 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:21:39.684243 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.684244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.684259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.684323 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:39.684478 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.685047 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.685784 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.685968 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.686030 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.686204 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:39.686290 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:39.686362 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.686410 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.686418 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:21:39.688205 3363 raft_consensus.cc:1261] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 1 FOLLOWER]: Refusing update from remote peer fbf19c105cf74eb0a9f78955694b0a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:39.688232 3497 raft_consensus.cc:1261] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 1 FOLLOWER]: Refusing update from remote peer fbf19c105cf74eb0a9f78955694b0a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:39.688557 3702 consensus_queue.cc:1035] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e3d37d073f469d9ab5c5ea8c1ecccb" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:39.688716 3704 consensus_queue.cc:1035] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76384980e4134b21b4ecec653221b40b" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:39.690990 3707 mvcc.cc:205] Tried to move back new op lower bound from 7333803211520339968 to 7333803211484319744. Current Snapshot: MvccSnapshot[applied={T|T < 7333803211520339968}]
I20260927 03:21:39.691648 3709 mvcc.cc:205] Tried to move back new op lower bound from 7333803211520339968 to 7333803211484319744. Current Snapshot: MvccSnapshot[applied={T|T < 7333803211520339968}]
I20260927 03:21:39.692154 3708 mvcc.cc:205] Tried to move back new op lower bound from 7333803211520339968 to 7333803211484319744. Current Snapshot: MvccSnapshot[applied={T|T < 7333803211520339968 or (T in {7333803211520339968})}]
I20260927 03:21:39.693338 3217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:39.693418 3217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:39.694145 3217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00cb84b6c25e40dfb5c510a0ec2d1a1a: Sending DeleteTablet for 3 replicas of tablet 5a25e08af965441799d5398413c7b602
I20260927 03:21:39.694455 3343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a25e08af965441799d5398413c7b602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:39 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:36420
I20260927 03:21:39.694458 3477 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a25e08af965441799d5398413c7b602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:39 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:39702
I20260927 03:21:39.694757 3611 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a25e08af965441799d5398413c7b602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:39 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:38230
I20260927 03:21:39.694972 3719 tablet_replica.cc:330] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: stopping tablet replica
I20260927 03:21:39.695048 3718 tablet_replica.cc:330] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb: stopping tablet replica
I20260927 03:21:39.695106 3719 raft_consensus.cc:2229] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:21:39.695149 3718 raft_consensus.cc:2229] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:39.695252 3718 raft_consensus.cc:2258] T 5a25e08af965441799d5398413c7b602 P a9e3d37d073f469d9ab5c5ea8c1ecccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:39.695250 3719 raft_consensus.cc:2258] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:39.695398 3717 tablet_replica.cc:330] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b: stopping tablet replica
I20260927 03:21:39.695472 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3280
I20260927 03:21:39.695492 3717 raft_consensus.cc:2229] T 5a25e08af965441799d5398413c7b602 P 76384980e4134b21b4ecec653221b40b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:39.700704 3719 ts_tablet_manager.cc:1905] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:21:39.701862 3203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:42249 (error 108)
I20260927 03:21:39.702227 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3414
I20260927 03:21:39.703250 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.703113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:42249 (local address 127.0.0.1:36426)
0927 03:21:39.703171 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.703208 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:21:39.703365 3203 catalog_manager.cc:4584] TS 76384980e4134b21b4ecec653221b40b (127.17.52.129:42249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a25e08af965441799d5398413c7b602: Network error: Client connection negotiation failed: client connection to 127.17.52.129:42249: connect: Connection refused (error 111)
I20260927 03:21:39.703580 3719 ts_tablet_manager.cc:1918] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:39.703647 3719 log.cc:1196] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/5a25e08af965441799d5398413c7b602
I20260927 03:21:39.703914 3719 ts_tablet_manager.cc:1939] T 5a25e08af965441799d5398413c7b602 P fbf19c105cf74eb0a9f78955694b0a02: Deleting consensus metadata
I20260927 03:21:39.704422 3204 catalog_manager.cc:4841] TS fbf19c105cf74eb0a9f78955694b0a02 (127.17.52.131:37489): tablet 5a25e08af965441799d5398413c7b602 (table test-table [id=679f2fb9884e42378280df3e52d12b3b]) successfully deleted
I20260927 03:21:39.708335 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3548
I20260927 03:21:39.708467 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.708323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:43899 (local address 127.0.0.1:39708)
0927 03:21:39.708400 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:39.708436 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.130:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 03:21:39.713217 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3186
2026-09-27T03:21:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T03:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:39Z Disabled control of system clock
I20260927 03:21:39.741075 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40397
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:39.820657 3724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:39.820894 3724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:39.820919 3724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:39.822002 3724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:39.822036 3724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:39.822049 3724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:39.822062 3724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:39.822073 3724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:39.823313 3724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:40397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:39.823493 3724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:39.823717 3724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:39.828532 3730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.828514 3729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.828698 3724 server_base.cc:1034] running on GCE node
W20260927 03:21:39.828506 3732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.829035 3724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:39.829335 3724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:39.830534 3724 hybrid_clock.cc:648] HybridClock initialized: now 1790479299830531 us; error 35 us; skew 500 ppm
I20260927 03:21:39.831899 3724 webserver.cc:460] Webserver started at http://127.17.52.190:40183/ using document root <none> and password file <none>
I20260927 03:21:39.832152 3724 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:39.832227 3724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:39.832376 3724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:39.833523 3724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "9ad7cf656d88474fa8da0531056b73dc"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1a4028ebadd97cb2e795b4f6f7fab0a7"
server_key_iv: "47811466bb9341b91404484a05bca7a9"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.833877 3724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "9ad7cf656d88474fa8da0531056b73dc"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1a4028ebadd97cb2e795b4f6f7fab0a7"
server_key_iv: "47811466bb9341b91404484a05bca7a9"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.835212 3724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:21:39.835937 3738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:39.836114 3724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:39.836220 3724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "9ad7cf656d88474fa8da0531056b73dc"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "1a4028ebadd97cb2e795b4f6f7fab0a7"
server_key_iv: "47811466bb9341b91404484a05bca7a9"
server_key_version: "encryptionkey@0"
I20260927 03:21:39.836302 3724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:39.854372 3724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:39.854681 3724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:39.854828 3724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:39.858945 3724 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:40397
I20260927 03:21:39.858976 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:40397 every 8 connection(s)
I20260927 03:21:39.859314 3724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:39.860249 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:39.862402 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: Bootstrap starting.
I20260927 03:21:39.862920 3791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:39.863160 3791 log.cc:824] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:39.863750 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: No bootstrap required, opened a new log
I20260927 03:21:39.865338 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3724
I20260927 03:21:39.865432 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:39.865412 3791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } }
I20260927 03:21:39.865597 3791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:39.865648 3791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad7cf656d88474fa8da0531056b73dc, State: Initialized, Role: FOLLOWER
I20260927 03:21:39.865669 17618 external_mini_cluster.cc:1428] Setting key 306a02c187f35698cdbf9edcddd09a8d
I20260927 03:21:39.865839 3791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } }
I20260927 03:21:39.865943 3791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:39.865979 3791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:39.866035 3791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:39.866703 3791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } }
I20260927 03:21:39.866814 3791 leader_election.cc:302] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ad7cf656d88474fa8da0531056b73dc; no voters:
I20260927 03:21:39.866977 3791 leader_election.cc:288] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:39.867053 3796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:39.867211 3791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:39.867234 3796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [term 1 LEADER]: Becoming Leader. State: Replica: 9ad7cf656d88474fa8da0531056b73dc, State: Running, Role: LEADER
I20260927 03:21:39.867319 3796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } }
I20260927 03:21:39.867702 3796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ad7cf656d88474fa8da0531056b73dc. Latest consensus state: current_term: 1 leader_uuid: "9ad7cf656d88474fa8da0531056b73dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } } }
I20260927 03:21:39.867775 3796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:39.867713 3797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ad7cf656d88474fa8da0531056b73dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad7cf656d88474fa8da0531056b73dc" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40397 } } }
I20260927 03:21:39.867821 3797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:39.868534 3805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:39.869223 3805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:39.870702 3805 catalog_manager.cc:1347] Generated new cluster ID: a977f37ecdde4eea9cc19538f925bfba
I20260927 03:21:39.870754 3805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:39.872401 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:39.866222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.17.52.190:40397)
0927 03:21:39.866471 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:39.866480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:39.866514 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:21:39.866632 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:39.866639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:39.866725 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:39.866877 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:39.867188 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.869100 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.870050 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:39.870649 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:39.870765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:39.871673 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:39.871677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:39.871678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:39.871874 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:39.871913 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:39.871926 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:39.871986 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:39.872088 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:21:39.872161 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20260927 03:21:39.876613 3805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:39.877276 3805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:39.903424 3805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: Generated new TSK 0
I20260927 03:21:39.903661 3805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:39.907517 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--builtin_ntp_servers=127.17.52.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:39.988718 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:39.988924 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:39.988960 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:39.990149 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:39.990213 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:39.990255 3815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:39.991564 3815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:39.991833 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:39.992106 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:39.996961 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:39.996960 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.997193 3815 server_base.cc:1034] running on GCE node
W20260927 03:21:39.996934 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:39.997424 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:39.997680 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:39.998853 3815 hybrid_clock.cc:648] HybridClock initialized: now 1790479299998834 us; error 34 us; skew 500 ppm
I20260927 03:21:40.000043 3815 webserver.cc:460] Webserver started at http://127.17.52.129:41857/ using document root <none> and password file <none>
I20260927 03:21:40.000247 3815 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:40.000288 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:40.000377 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:40.001442 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "69dd37b7eb7a40739e718b0d5767311c"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "6e27ac09434c40304e9cbae6ca4515b0"
server_key_iv: "543e308157091f4e2783ef7f92a40fec"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.001740 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "69dd37b7eb7a40739e718b0d5767311c"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "6e27ac09434c40304e9cbae6ca4515b0"
server_key_iv: "543e308157091f4e2783ef7f92a40fec"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.003067 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:21:40.003782 3829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.004007 3815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:40.004099 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "69dd37b7eb7a40739e718b0d5767311c"
format_stamp: "Formatted at 2026-09-27 03:21:39 on dist-test-slave-7j43"
server_key: "6e27ac09434c40304e9cbae6ca4515b0"
server_key_iv: "543e308157091f4e2783ef7f92a40fec"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.004173 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:40.016086 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:40.016396 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:40.016546 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:40.016813 3815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:40.017122 3815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:40.017154 3815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.017213 3815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:40.017242 3815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.022976 3815 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:44829
I20260927 03:21:40.023010 3942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:44829 every 8 connection(s)
I20260927 03:21:40.023330 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:40.028291 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.024045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:35763 (local address 127.17.52.190:40397)
0927 03:21:40.024107 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.024109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.024327 (+ 218us) server_negotiation.cc:407] Connection header received
0927 03:21:40.024943 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.024944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.024958 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.024997 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:40.025626 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.026147 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.027092 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.027295 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.027352 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.028049 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:40.028050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:40.028051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:40.028055 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:40.028068 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:40.028072 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:40.028127 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.028207 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.028234 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:21:40.028503 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.024022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40397 (local address 127.17.52.129:35763)
0927 03:21:40.024215 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.024246 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.024822 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.025066 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.025069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.025077 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:40.025550 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.025554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.026224 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.026225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.027010 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.027012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.027373 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.027374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.027477 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.027913 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:40.027913 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:40.027939 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:40.027960 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:40.028136 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:40.028140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:40.028142 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.028192 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.028264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260927 03:21:40.029125 3943 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40397
I20260927 03:21:40.029248 3943 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:40.029443 3943 heartbeater.cc:511] Master 127.17.52.190:40397 requested a full tablet report, sending...
I20260927 03:21:40.029928 3755 ts_manager.cc:194] Registered new tserver with Master: 69dd37b7eb7a40739e718b0d5767311c (127.17.52.129:44829)
I20260927 03:21:40.030483 3755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.129:35763
I20260927 03:21:40.032385 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3815
I20260927 03:21:40.032541 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:40.032765 17618 external_mini_cluster.cc:1428] Setting key 440d862369666a1a64b690cce06f3f9a
I20260927 03:21:40.033560 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--builtin_ntp_servers=127.17.52.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:40.114425 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:40.114621 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:40.114641 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:40.115757 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:40.115797 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:40.115823 3946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:40.117234 3946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:40.117460 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:40.117746 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:40.122588 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:40.122575 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:40.122786 3946 server_base.cc:1034] running on GCE node
W20260927 03:21:40.122575 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:40.123044 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:40.123314 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:40.124469 3946 hybrid_clock.cc:648] HybridClock initialized: now 1790479300124455 us; error 28 us; skew 500 ppm
I20260927 03:21:40.125888 3946 webserver.cc:460] Webserver started at http://127.17.52.130:40529/ using document root <none> and password file <none>
I20260927 03:21:40.126143 3946 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:40.126199 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:40.126368 3946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:40.127453 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "899a086af4cf4effbac3a8f771a78439"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "cbc9fd17a4e1ceebb4793b569e880199"
server_key_iv: "f41d23aa6c15f288c87051cceb906171"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.127815 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "899a086af4cf4effbac3a8f771a78439"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "cbc9fd17a4e1ceebb4793b569e880199"
server_key_iv: "f41d23aa6c15f288c87051cceb906171"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.129227 3946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:21:40.129985 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.130160 3946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:40.130262 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "899a086af4cf4effbac3a8f771a78439"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "cbc9fd17a4e1ceebb4793b569e880199"
server_key_iv: "f41d23aa6c15f288c87051cceb906171"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.130344 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:40.148140 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:40.148468 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:40.148658 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:40.148891 3946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:40.149228 3946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:40.149286 3946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.149327 3946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:40.149355 3946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.155123 3946 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:46507
I20260927 03:21:40.155165 4073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:46507 every 8 connection(s)
I20260927 03:21:40.155493 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:40.158418 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 3946
I20260927 03:21:40.158541 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:40.158763 17618 external_mini_cluster.cc:1428] Setting key e1e3d73d8ecbe4c19e53117cb4a22bb3
I20260927 03:21:40.159714 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--builtin_ntp_servers=127.17.52.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:21:40.161195 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.156194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:43979 (local address 127.17.52.190:40397)
0927 03:21:40.156258 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.156260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.156504 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:21:40.157063 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.157064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.157075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.157137 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:40.157792 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.158303 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.159640 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.159918 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.159990 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.160905 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:40.160907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:40.160907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:40.160912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:40.160926 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:40.160930 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:40.160978 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.161079 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.161108 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:21:40.161413 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.156184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40397 (local address 127.17.52.130:43979)
0927 03:21:40.156374 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.156414 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.156961 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.157190 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.157193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.157201 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:40.157684 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.157688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.158459 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.158461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.159553 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.159557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.160001 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.160002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.160134 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.160739 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:40.160739 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:40.160769 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:40.160793 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:40.160977 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:40.160982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:40.160985 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.161069 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.161176 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260927 03:21:40.161873 4074 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40397
I20260927 03:21:40.161967 4074 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:40.162163 4074 heartbeater.cc:511] Master 127.17.52.190:40397 requested a full tablet report, sending...
I20260927 03:21:40.162559 3755 ts_manager.cc:194] Registered new tserver with Master: 899a086af4cf4effbac3a8f771a78439 (127.17.52.130:46507)
I20260927 03:21:40.162933 3755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.130:43979
W20260927 03:21:40.241344 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:40.241549 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:40.241570 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:40.242753 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:40.242795 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:40.242820 4077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:40.244175 4077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:40.244428 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:40.244797 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:40.249667 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:40.249671 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:40.249828 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:40.249917 4077 server_base.cc:1034] running on GCE node
I20260927 03:21:40.250139 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:40.250378 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:40.251521 4077 hybrid_clock.cc:648] HybridClock initialized: now 1790479300251501 us; error 39 us; skew 500 ppm
I20260927 03:21:40.252777 4077 webserver.cc:460] Webserver started at http://127.17.52.131:33201/ using document root <none> and password file <none>
I20260927 03:21:40.253036 4077 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:40.253104 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:40.253249 4077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:40.254279 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "83d9532d8339b48bea3f5b6059ab7139"
server_key_iv: "24c0f944f13aae9a620cab23ee79806e"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.254649 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "83d9532d8339b48bea3f5b6059ab7139"
server_key_iv: "24c0f944f13aae9a620cab23ee79806e"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.255962 4077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:40.256848 4091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.257025 4077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:40.257126 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d"
format_stamp: "Formatted at 2026-09-27 03:21:40 on dist-test-slave-7j43"
server_key: "83d9532d8339b48bea3f5b6059ab7139"
server_key_iv: "24c0f944f13aae9a620cab23ee79806e"
server_key_version: "encryptionkey@0"
I20260927 03:21:40.257205 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:40.265370 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:40.265636 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:40.265785 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:40.266011 4077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:40.266324 4077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:40.266379 4077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.266420 4077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:40.266446 4077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:40.272387 4077 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:45645
I20260927 03:21:40.272431 4204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:45645 every 8 connection(s)
I20260927 03:21:40.272791 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:40.275209 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4077
I20260927 03:21:40.275295 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:40.275482 17618 external_mini_cluster.cc:1428] Setting key a9f37907a9139ea1c015714a73815b13
I20260927 03:21:40.278486 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.273514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:38137 (local address 127.17.52.190:40397)
0927 03:21:40.273585 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.273587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.273887 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:21:40.274426 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.274428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.274439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.274490 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:40.275223 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.276021 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.277179 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.277388 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.277448 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.278228 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:40.278229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:40.278229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:40.278233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:40.278248 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:40.278253 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:40.278301 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.278392 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.278417 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:21:40.278653 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.273522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40397 (local address 127.17.52.131:38137)
0927 03:21:40.273733 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.273800 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.274333 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.274547 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.274551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.274560 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:40.275110 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.275115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.276117 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.276118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.277104 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.277107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.277467 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.277468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.277572 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.278085 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:40.278086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:40.278113 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:40.278135 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:40.278322 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:40.278325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:40.278327 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.278377 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.278457 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 03:21:40.279137 4205 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40397
I20260927 03:21:40.279238 4205 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:40.279439 4205 heartbeater.cc:511] Master 127.17.52.190:40397 requested a full tablet report, sending...
I20260927 03:21:40.279850 3755 ts_manager.cc:194] Registered new tserver with Master: b51e72ca50bf49108f4cb9cde5b3fa3d (127.17.52.131:45645)
I20260927 03:21:40.280248 3755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.131:38137
I20260927 03:21:40.287276 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:40.292124 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.288929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.17.52.190:40397)
0927 03:21:40.289020 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.289022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.289114 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:21:40.289188 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.289189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.289203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.289256 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:40.289463 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.289991 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.290765 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.290969 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.291063 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.291812 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:40.291814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:40.291814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:40.291820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:40.291836 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:40.291840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:40.291957 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.292029 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.292040 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:21:40.293699 3755 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:40.295229 3755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:40.301698 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.17.52.131:45645)
0927 03:21:40.297624 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.297629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.298065 (+ 436us) server_negotiation.cc:407] Connection header received
0927 03:21:40.298110 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.298112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.298127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.298197 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:40.298372 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299449 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.300776 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301275 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301480 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.301565 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.301605 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260927 03:21:40.301684 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:45645 (local address 127.0.0.1:37758)
0927 03:21:40.297718 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.297749 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.297861 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.298261 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.298262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.298264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:40.298305 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.298306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299529 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.299530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.300683 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.300686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301391 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.301470 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301483 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.301553 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.301605 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":28,"thread_start_us":90,"threads_started":1}
I20260927 03:21:40.301817 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.17.52.130:46507)
0927 03:21:40.297662 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.297666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.297829 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:21:40.297924 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.297926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.297944 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.298030 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:40.298218 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299392 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.300564 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301310 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301478 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.301696 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.301736 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260927 03:21:40.301868 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.17.52.129:44829)
0927 03:21:40.298151 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.298155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.298164 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:40.298200 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.298201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.298214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.298282 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:40.298450 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299142 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.300657 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301345 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301403 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301478 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.301743 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.301784 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":98,"threads_started":1}
I20260927 03:21:40.302285 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:44829 (local address 127.0.0.1:60502)
0927 03:21:40.297956 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.297965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.298004 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.298359 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.298359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.298361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:40.298391 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.298392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299246 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.299247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.300527 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.300530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301514 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.301597 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301605 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.301654 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.301703 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260927 03:21:40.302513 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.297385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:46507 (local address 127.0.0.1:60162)
0927 03:21:40.297688 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.297745 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.297825 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.298039 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.298042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.298044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:40.298115 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.298117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.299498 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.299499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.300476 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.300478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.301516 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.301517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.301597 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.301603 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.302127 (+ 524us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.302162 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":66,"thread_start_us":52,"threads_started":1}
I20260927 03:21:40.302436 4139 tablet_service.cc:1467] Processing CreateTablet for tablet b226814a35ff46049d015dd26363b8ec (DEFAULT_TABLE table=test-table [id=e17a300a6ae64aacb87e4ec9c9b686ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:40.302834 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b226814a35ff46049d015dd26363b8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:40.303083 3877 tablet_service.cc:1467] Processing CreateTablet for tablet b226814a35ff46049d015dd26363b8ec (DEFAULT_TABLE table=test-table [id=e17a300a6ae64aacb87e4ec9c9b686ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:40.303153 4008 tablet_service.cc:1467] Processing CreateTablet for tablet b226814a35ff46049d015dd26363b8ec (DEFAULT_TABLE table=test-table [id=e17a300a6ae64aacb87e4ec9c9b686ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:40.303426 3877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b226814a35ff46049d015dd26363b8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:40.303426 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b226814a35ff46049d015dd26363b8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:40.305572 4219 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Bootstrap starting.
I20260927 03:21:40.306191 4219 tablet_bootstrap.cc:654] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:40.306494 4219 log.cc:824] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:40.306722 4220 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Bootstrap starting.
I20260927 03:21:40.307327 4220 tablet_bootstrap.cc:654] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:40.307601 4220 log.cc:824] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:40.307862 4221 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Bootstrap starting.
I20260927 03:21:40.308283 4220 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: No bootstrap required, opened a new log
I20260927 03:21:40.308339 4220 ts_tablet_manager.cc:1397] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:40.308419 4221 tablet_bootstrap.cc:654] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:40.308818 4221 log.cc:824] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:40.309242 4219 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: No bootstrap required, opened a new log
I20260927 03:21:40.309340 4219 ts_tablet_manager.cc:1397] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:21:40.309535 4221 tablet_bootstrap.cc:492] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: No bootstrap required, opened a new log
I20260927 03:21:40.309615 4221 ts_tablet_manager.cc:1397] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:21:40.311003 4220 raft_consensus.cc:348] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.311167 4220 raft_consensus.cc:374] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:40.311206 4220 raft_consensus.cc:729] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899a086af4cf4effbac3a8f771a78439, State: Initialized, Role: FOLLOWER
I20260927 03:21:40.311321 4220 consensus_queue.cc:260] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.311564 4220 ts_tablet_manager.cc:1428] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:40.311695 4074 heartbeater.cc:503] Master 127.17.52.190:40397 was elected leader, sending a full tablet report...
I20260927 03:21:40.311971 4219 raft_consensus.cc:348] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.312084 4219 raft_consensus.cc:374] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:40.312104 4219 raft_consensus.cc:729] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51e72ca50bf49108f4cb9cde5b3fa3d, State: Initialized, Role: FOLLOWER
I20260927 03:21:40.312105 4221 raft_consensus.cc:348] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.312211 4219 consensus_queue.cc:260] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.312245 4221 raft_consensus.cc:374] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:40.312276 4221 raft_consensus.cc:729] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dd37b7eb7a40739e718b0d5767311c, State: Initialized, Role: FOLLOWER
I20260927 03:21:40.312458 4219 ts_tablet_manager.cc:1428] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:40.312431 4221 consensus_queue.cc:260] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.312487 4205 heartbeater.cc:503] Master 127.17.52.190:40397 was elected leader, sending a full tablet report...
I20260927 03:21:40.312716 4221 ts_tablet_manager.cc:1428] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:21:40.312783 3943 heartbeater.cc:503] Master 127.17.52.190:40397 was elected leader, sending a full tablet report...
W20260927 03:21:40.406030 4075 tablet.cc:2369] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:21:40.523329 4206 tablet.cc:2369] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:21:40.524021 3944 tablet.cc:2369] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:21:40.563239 4227 raft_consensus.cc:482] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:40.563380 4227 raft_consensus.cc:504] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.563723 4227 leader_election.cc:288] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b51e72ca50bf49108f4cb9cde5b3fa3d (127.17.52.131:45645), 899a086af4cf4effbac3a8f771a78439 (127.17.52.130:46507)
I20260927 03:21:40.566644 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.563972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:50539 (local address 127.17.52.130:46507)
0927 03:21:40.564025 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.564027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.564116 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:21:40.564202 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.564203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.564212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.564292 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:40.564455 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.564980 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.565920 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.566275 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.566363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.566371 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.566527 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.566570 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:21:40.566650 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.563970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:48191 (local address 127.17.52.131:45645)
0927 03:21:40.564015 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.564017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.564127 (+ 110us) server_negotiation.cc:407] Connection header received
0927 03:21:40.564210 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.564211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.564220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.564266 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:40.564428 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.564938 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.565900 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.566275 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.566363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.566371 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.566543 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.566582 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:21:40.566671 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.563908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:46507 (local address 127.17.52.129:50539)
0927 03:21:40.564029 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.564039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.564132 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.564294 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.564295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.564297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:40.564383 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.564385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.565069 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.565070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.565814 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.565816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.566363 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.566364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.566452 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.566455 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.566522 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.566567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260927 03:21:40.566679 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.563914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:45645 (local address 127.17.52.129:48191)
0927 03:21:40.564048 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.564058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:40.564139 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:40.564278 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:40.564279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:40.564281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:40.564358 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.564360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.565027 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.565028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.565806 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:40.565808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.566361 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.566362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:40.566454 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.566458 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:40.566533 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:21:40.566576 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":23,"thread_start_us":62,"threads_started":1}
I20260927 03:21:40.567094 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b226814a35ff46049d015dd26363b8ec" candidate_uuid: "69dd37b7eb7a40739e718b0d5767311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899a086af4cf4effbac3a8f771a78439" is_pre_election: true
I20260927 03:21:40.567283 4028 raft_consensus.cc:2454] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dd37b7eb7a40739e718b0d5767311c in term 0.
I20260927 03:21:40.567332 4159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b226814a35ff46049d015dd26363b8ec" candidate_uuid: "69dd37b7eb7a40739e718b0d5767311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" is_pre_election: true
I20260927 03:21:40.567500 4159 raft_consensus.cc:2454] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dd37b7eb7a40739e718b0d5767311c in term 0.
I20260927 03:21:40.567539 3831 leader_election.cc:302] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dd37b7eb7a40739e718b0d5767311c, 899a086af4cf4effbac3a8f771a78439; no voters:
I20260927 03:21:40.567677 4227 raft_consensus.cc:2781] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:40.567739 4227 raft_consensus.cc:482] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:40.567763 4227 raft_consensus.cc:3037] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:40.568660 4227 raft_consensus.cc:504] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.568795 4227 leader_election.cc:288] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [CANDIDATE]: Term 1 election: Requested vote from peers b51e72ca50bf49108f4cb9cde5b3fa3d (127.17.52.131:45645), 899a086af4cf4effbac3a8f771a78439 (127.17.52.130:46507)
I20260927 03:21:40.568933 4159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b226814a35ff46049d015dd26363b8ec" candidate_uuid: "69dd37b7eb7a40739e718b0d5767311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d"
I20260927 03:21:40.568967 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b226814a35ff46049d015dd26363b8ec" candidate_uuid: "69dd37b7eb7a40739e718b0d5767311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899a086af4cf4effbac3a8f771a78439"
I20260927 03:21:40.569016 4159 raft_consensus.cc:3037] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:40.569036 4028 raft_consensus.cc:3037] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:40.569742 4028 raft_consensus.cc:2454] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dd37b7eb7a40739e718b0d5767311c in term 1.
I20260927 03:21:40.569794 4159 raft_consensus.cc:2454] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dd37b7eb7a40739e718b0d5767311c in term 1.
I20260927 03:21:40.569959 3833 leader_election.cc:302] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dd37b7eb7a40739e718b0d5767311c, b51e72ca50bf49108f4cb9cde5b3fa3d; no voters:
I20260927 03:21:40.570060 4227 raft_consensus.cc:2781] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:40.570199 4227 raft_consensus.cc:686] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [term 1 LEADER]: Becoming Leader. State: Replica: 69dd37b7eb7a40739e718b0d5767311c, State: Running, Role: LEADER
I20260927 03:21:40.570312 4227 consensus_queue.cc:237] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } }
I20260927 03:21:40.571100 3754 catalog_manager.cc:5495] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c reported cstate change: term changed from 0 to 1, leader changed from <none> to 69dd37b7eb7a40739e718b0d5767311c (127.17.52.129). New cstate: current_term: 1 leader_uuid: "69dd37b7eb7a40739e718b0d5767311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dd37b7eb7a40739e718b0d5767311c" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 44829 } health_report { overall_health: HEALTHY } } }
I20260927 03:21:40.585789 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.583179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.17.52.129:44829)
0927 03:21:40.583223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:40.583225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:40.583234 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:40.583293 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:40.583294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:40.583314 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:40.583405 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:40.583545 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.584016 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.584901 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:40.585104 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:40.585185 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:40.585344 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:40.585474 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:40.585576 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:40.585694 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:21:40.585705 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:21:40.587677 4159 raft_consensus.cc:1261] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 1 FOLLOWER]: Refusing update from remote peer 69dd37b7eb7a40739e718b0d5767311c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:40.587666 4028 raft_consensus.cc:1261] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 1 FOLLOWER]: Refusing update from remote peer 69dd37b7eb7a40739e718b0d5767311c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:40.588070 4227 consensus_queue.cc:1035] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "899a086af4cf4effbac3a8f771a78439" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:40.588269 4230 consensus_queue.cc:1035] T b226814a35ff46049d015dd26363b8ec P 69dd37b7eb7a40739e718b0d5767311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51e72ca50bf49108f4cb9cde5b3fa3d" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:40.591691 4233 mvcc.cc:205] Tried to move back new op lower bound from 7333803215204229120 to 7333803215136645120. Current Snapshot: MvccSnapshot[applied={T|T < 7333803215204229120 or (T in {7333803215204229120})}]
I20260927 03:21:40.592912 3755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:40.593008 3755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:40.593744 3755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ad7cf656d88474fa8da0531056b73dc: Sending DeleteTablet for 3 replicas of tablet b226814a35ff46049d015dd26363b8ec
I20260927 03:21:40.594120 4008 tablet_service.cc:1514] Processing DeleteTablet for tablet b226814a35ff46049d015dd26363b8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:40 UTC) from {username='slave'} at 127.0.0.1:60162
I20260927 03:21:40.594172 3877 tablet_service.cc:1514] Processing DeleteTablet for tablet b226814a35ff46049d015dd26363b8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:40 UTC) from {username='slave'} at 127.0.0.1:60502
I20260927 03:21:40.594347 4243 tablet_replica.cc:330] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: stopping tablet replica
I20260927 03:21:40.594367 4139 tablet_service.cc:1514] Processing DeleteTablet for tablet b226814a35ff46049d015dd26363b8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:40 UTC) from {username='slave'} at 127.0.0.1:37758
I20260927 03:21:40.594449 4243 raft_consensus.cc:2229] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:40.594553 4243 raft_consensus.cc:2258] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:40.594571 4245 tablet_replica.cc:330] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: stopping tablet replica
I20260927 03:21:40.594674 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3815
I20260927 03:21:40.594718 4245 raft_consensus.cc:2229] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:40.595515 4245 raft_consensus.cc:2258] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:40.595788 4243 ts_tablet_manager.cc:1905] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:40.595919 4245 ts_tablet_manager.cc:1905] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:40.597456 4243 ts_tablet_manager.cc:1918] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:40.597570 4243 log.cc:1196] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/b226814a35ff46049d015dd26363b8ec
I20260927 03:21:40.597849 4243 ts_tablet_manager.cc:1939] T b226814a35ff46049d015dd26363b8ec P 899a086af4cf4effbac3a8f771a78439: Deleting consensus metadata
I20260927 03:21:40.597864 4245 ts_tablet_manager.cc:1918] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:40.597927 4245 log.cc:1196] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/b226814a35ff46049d015dd26363b8ec
I20260927 03:21:40.598169 4245 ts_tablet_manager.cc:1939] T b226814a35ff46049d015dd26363b8ec P b51e72ca50bf49108f4cb9cde5b3fa3d: Deleting consensus metadata
I20260927 03:21:40.598260 3740 catalog_manager.cc:4841] TS 899a086af4cf4effbac3a8f771a78439 (127.17.52.130:46507): tablet b226814a35ff46049d015dd26363b8ec (table test-table [id=e17a300a6ae64aacb87e4ec9c9b686ca]) successfully deleted
I20260927 03:21:40.598542 3742 catalog_manager.cc:4841] TS b51e72ca50bf49108f4cb9cde5b3fa3d (127.17.52.131:45645): tablet b226814a35ff46049d015dd26363b8ec (table test-table [id=e17a300a6ae64aacb87e4ec9c9b686ca]) successfully deleted
W20260927 03:21:40.601711 3742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:44829 (error 108)
I20260927 03:21:40.602231 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:40.602043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:44829 (local address 127.0.0.1:60524)
0927 03:21:40.602115 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:40.602191 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:44829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 03:21:40.602246 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3946
W20260927 03:21:40.602380 3742 catalog_manager.cc:4584] TS 69dd37b7eb7a40739e718b0d5767311c (127.17.52.129:44829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b226814a35ff46049d015dd26363b8ec: Network error: Client connection negotiation failed: client connection to 127.17.52.129:44829: connect: Connection refused (error 111)
I20260927 03:21:40.606735 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4077
I20260927 03:21:40.611642 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 3724
2026-09-27T03:21:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:40 dist-test-slave-7j43 krb5kdc[4249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:40 dist-test-slave-7j43 krb5kdc[4249](info): set up 2 sockets
Sep 27 03:21:40 dist-test-slave-7j43 krb5kdc[4249](info): commencing operation
krb5kdc: starting...
W20260927 03:21:41.649353 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:41 dist-test-slave-7j43 krb5kdc[4249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:21:41.764233 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45747
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:45747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:41.845609 4262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:41.845839 4262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:41.845862 4262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:41.846920 4262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:41.846956 4262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:41.846973 4262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:41.846987 4262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:41.847002 4262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:41.848371 4262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:45747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:41.848556 4262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:41.848821 4262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:41.853641 4270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:41.853693 4267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:41.853659 4268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:41.853693 4262 server_base.cc:1034] running on GCE node
I20260927 03:21:41.854094 4262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:41.854386 4262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:41.855545 4262 hybrid_clock.cc:648] HybridClock initialized: now 1790479301855528 us; error 31 us; skew 500 ppm
Sep 27 03:21:41 dist-test-slave-7j43 krb5kdc[4249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:41.862322 4262 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:21:41.862887 4262 webserver.cc:460] Webserver started at http://127.17.52.190:43701/ using document root <none> and password file <none>
I20260927 03:21:41.863125 4262 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:41.863216 4262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:41.863368 4262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:41.864450 4262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "db57118535eb4cc7a0081a6d7dbd01ef"
format_stamp: "Formatted at 2026-09-27 03:21:41 on dist-test-slave-7j43"
server_key: "8f286bda7d9e98c8ff41f0f34278f578"
server_key_iv: "b65d08a23673ae218e662cbfc36c76fa"
server_key_version: "encryptionkey@0"
I20260927 03:21:41.864892 4262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "db57118535eb4cc7a0081a6d7dbd01ef"
format_stamp: "Formatted at 2026-09-27 03:21:41 on dist-test-slave-7j43"
server_key: "8f286bda7d9e98c8ff41f0f34278f578"
server_key_iv: "b65d08a23673ae218e662cbfc36c76fa"
server_key_version: "encryptionkey@0"
I20260927 03:21:41.866240 4262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:41.867002 4277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:41.867213 4262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:41.867333 4262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "db57118535eb4cc7a0081a6d7dbd01ef"
format_stamp: "Formatted at 2026-09-27 03:21:41 on dist-test-slave-7j43"
server_key: "8f286bda7d9e98c8ff41f0f34278f578"
server_key_iv: "b65d08a23673ae218e662cbfc36c76fa"
server_key_version: "encryptionkey@0"
I20260927 03:21:41.867410 4262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:41.874441 4262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:41.876137 4262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:41.876291 4262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:41.880343 4262 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:45747
I20260927 03:21:41.880369 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:45747 every 8 connection(s)
I20260927 03:21:41.880761 4262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:41.881583 4330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:41.883708 4330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: Bootstrap starting.
I20260927 03:21:41.884228 4330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:41.884462 4330 log.cc:824] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:41.885248 4330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: No bootstrap required, opened a new log
I20260927 03:21:41.886754 4330 raft_consensus.cc:348] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } }
I20260927 03:21:41.886884 4330 raft_consensus.cc:374] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:41.886907 4330 raft_consensus.cc:729] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db57118535eb4cc7a0081a6d7dbd01ef, State: Initialized, Role: FOLLOWER
I20260927 03:21:41.887039 4330 consensus_queue.cc:260] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } }
I20260927 03:21:41.887118 4330 raft_consensus.cc:388] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:41.887141 4330 raft_consensus.cc:482] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:41.887169 4330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:41.887620 4330 raft_consensus.cc:504] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } }
I20260927 03:21:41.887701 4330 leader_election.cc:302] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db57118535eb4cc7a0081a6d7dbd01ef; no voters:
I20260927 03:21:41.887820 4330 leader_election.cc:288] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:41.887895 4333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:41.888041 4333 raft_consensus.cc:686] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [term 1 LEADER]: Becoming Leader. State: Replica: db57118535eb4cc7a0081a6d7dbd01ef, State: Running, Role: LEADER
I20260927 03:21:41.888078 4330 sys_catalog.cc:564] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:41.888123 4333 consensus_queue.cc:237] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } }
I20260927 03:21:41.888491 4334 sys_catalog.cc:455] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } } }
I20260927 03:21:41.888595 4334 sys_catalog.cc:458] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:41.888784 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4262
I20260927 03:21:41.888913 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:41.889144 17618 external_mini_cluster.cc:1428] Setting key a50241f057b4b2e2d56bdad96852df52
I20260927 03:21:41.889186 4341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:41.889570 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader db57118535eb4cc7a0081a6d7dbd01ef. Latest consensus state: current_term: 1 leader_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db57118535eb4cc7a0081a6d7dbd01ef" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 45747 } } }
I20260927 03:21:41.889715 4341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:41.889720 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:41.891249 4341 catalog_manager.cc:1347] Generated new cluster ID: 6fb236957b414c4c9bd58683178d1780
I20260927 03:21:41.891301 4341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:21:41 dist-test-slave-7j43 krb5kdc[4249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:41.898613 4341 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:41.899279 4341 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:41.900003 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:41.889783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.17.52.190:45747)
0927 03:21:41.890393 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:41.890401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:41.890435 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:21:41.890545 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:41.890552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:41.890607 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:41.890759 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:41.891537 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:41.892424 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:41.893229 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:41.893644 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:41.893766 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:41.896027 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:41.896030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:41.896031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:41.896043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:41.898350 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:41.898758 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:41.898761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:41.898761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:41.898860 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:41.899078 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:41.899079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:41.899079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:41.899296 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:41.899476 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:41.899603 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 03:21:41.899851 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":202,"threads_started":1}
I20260927 03:21:41.905560 4341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: Generated new TSK 0
I20260927 03:21:41.905731 4341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.52.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
Entry for principal kudu/127.17.52.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab.
I20260927 03:21:41.938993 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:42.029260 4356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.029525 4356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.029567 4356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.030807 4356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.030882 4356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.030930 4356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:42.032482 4356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.129.keytab
--principal=kudu/127.17.52.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.032778 4356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.033025 4356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.038829 4364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.038858 4362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.039093 4356 server_base.cc:1034] running on GCE node
W20260927 03:21:42.038858 4361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.039397 4356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.039639 4356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.040808 4356 hybrid_clock.cc:648] HybridClock initialized: now 1790479302040799 us; error 32 us; skew 500 ppm
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:42.043170 4356 init.cc:377] Logged in from keytab as kudu/127.17.52.129@KRBTEST.COM (short username kudu)
I20260927 03:21:42.043630 4356 webserver.cc:460] Webserver started at http://127.17.52.129:37787/ using document root <none> and password file <none>
I20260927 03:21:42.043869 4356 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.043941 4356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.044071 4356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.045218 4356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "be134bcdefd844bda5d47917e2263d39"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9d48e1a281819cf1465b0259265c609b"
server_key_iv: "125368db75c91741ffcd50f9d73140f7"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.045594 4356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "be134bcdefd844bda5d47917e2263d39"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9d48e1a281819cf1465b0259265c609b"
server_key_iv: "125368db75c91741ffcd50f9d73140f7"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.046939 4356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:21:42.047678 4371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.047870 4356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:42.047971 4356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "be134bcdefd844bda5d47917e2263d39"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9d48e1a281819cf1465b0259265c609b"
server_key_iv: "125368db75c91741ffcd50f9d73140f7"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.048058 4356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.060794 4356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.062595 4356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.062754 4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.062974 4356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:42.063280 4356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:42.063345 4356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.063387 4356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:42.063410 4356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.069255 4356 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:40343
I20260927 03:21:42.069305 4484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:40343 every 8 connection(s)
I20260927 03:21:42.069670 4356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:42.074882 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4356
I20260927 03:21:42.075017 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:42.075233 17618 external_mini_cluster.cc:1428] Setting key b762cb88ababb6db6c7128730c764ab1
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.129@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:42.079799 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.070405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:54363 (local address 127.17.52.190:45747)
0927 03:21:42.070447 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.070450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.070677 (+ 227us) server_negotiation.cc:407] Connection header received
0927 03:21:42.071272 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.071273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.071290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.071354 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.072275 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.072852 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.073846 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.074051 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.074139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.076531 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.076533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.076533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:42.076541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.078150 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.078710 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.078712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.078712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.078780 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.079068 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.079069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.079070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.079200 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.079292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.079587 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.079698 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:21:42.079933 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.070380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45747 (local address 127.17.52.129:54363)
0927 03:21:42.070554 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.070589 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.071167 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.071423 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.071427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.071982 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:42.072168 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.072173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.072942 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.072943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.073732 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.073734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.074170 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.074171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.074306 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.074803 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:42.074810 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:42.076445 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:42.078246 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.078256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.078258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.078602 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.078859 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.078860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.078860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.078989 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.079279 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:42.079286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:42.079430 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.079544 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.079702 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260927 03:21:42.080389 4485 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45747
I20260927 03:21:42.080499 4485 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:42.080731 4485 heartbeater.cc:511] Master 127.17.52.190:45747 requested a full tablet report, sending...
I20260927 03:21:42.081274 4294 ts_manager.cc:194] Registered new tserver with Master: be134bcdefd844bda5d47917e2263d39 (127.17.52.129:40343)
I20260927 03:21:42.081908 4294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.129@KRBTEST.COM'} at 127.17.52.129:54363
WARNING: no policy specified for kudu/127.17.52.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
Entry for principal kudu/127.17.52.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab.
I20260927 03:21:42.099598 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:42.178061 4490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.178243 4490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.178267 4490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.179313 4490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.179346 4490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.179370 4490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:42.180720 4490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.130.keytab
--principal=kudu/127.17.52.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.180899 4490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.181061 4490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.185770 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.185771 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.185811 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.186197 4490 server_base.cc:1034] running on GCE node
I20260927 03:21:42.186350 4490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.186570 4490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.187693 4490 hybrid_clock.cc:648] HybridClock initialized: now 1790479302187681 us; error 27 us; skew 500 ppm
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:42.190099 4490 init.cc:377] Logged in from keytab as kudu/127.17.52.130@KRBTEST.COM (short username kudu)
I20260927 03:21:42.190500 4490 webserver.cc:460] Webserver started at http://127.17.52.130:42307/ using document root <none> and password file <none>
I20260927 03:21:42.190732 4490 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.190811 4490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.190966 4490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.191952 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "fee568e9fa1e4fe084e8129f64edf725"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "48d45af1e63e8645284167f75d76da4c"
server_key_iv: "b8c221147d2ec8a80327f8ffb50285e9"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.192304 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "fee568e9fa1e4fe084e8129f64edf725"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "48d45af1e63e8645284167f75d76da4c"
server_key_iv: "b8c221147d2ec8a80327f8ffb50285e9"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.193698 4490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:42.194438 4505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.194625 4490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:42.194723 4490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "fee568e9fa1e4fe084e8129f64edf725"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "48d45af1e63e8645284167f75d76da4c"
server_key_iv: "b8c221147d2ec8a80327f8ffb50285e9"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.194808 4490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.207261 4490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.209082 4490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.209241 4490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.209468 4490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:42.209762 4490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:42.209820 4490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.209865 4490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:42.209893 4490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.215632 4490 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:40493
I20260927 03:21:42.215706 4618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:40493 every 8 connection(s)
I20260927 03:21:42.216024 4490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.130@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:42.223968 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4490
I20260927 03:21:42.224057 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:42.224301 17618 external_mini_cluster.cc:1428] Setting key 62fe70dbcc14ac6f026b4ddd775cf066
I20260927 03:21:42.225304 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.216757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:36257 (local address 127.17.52.190:45747)
0927 03:21:42.216808 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.216810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.216989 (+ 179us) server_negotiation.cc:407] Connection header received
0927 03:21:42.217525 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.217527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.217540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.217599 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.218557 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.219067 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.220062 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.220278 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.220355 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.222406 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.222408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.222409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:42.222417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.223675 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.224148 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.224150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.224150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.224220 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.224520 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.224521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.224521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.224687 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.224783 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.225096 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.225185 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:21:42.225374 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.216720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45747 (local address 127.17.52.130:36257)
0927 03:21:42.216856 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.216900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.217410 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.217678 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.217682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.218249 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:42.218433 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.218438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.219166 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.219167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.219963 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.219965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.220373 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.220374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.220500 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.221015 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:42.221023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:42.222307 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:42.223788 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.223796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.223798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.224043 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.224300 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.224300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.224301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.224431 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.224772 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:42.224779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:42.224912 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.225052 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.225190 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 03:21:42.226033 4619 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45747
I20260927 03:21:42.226190 4619 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:42.226392 4619 heartbeater.cc:511] Master 127.17.52.190:45747 requested a full tablet report, sending...
I20260927 03:21:42.226801 4294 ts_manager.cc:194] Registered new tserver with Master: fee568e9fa1e4fe084e8129f64edf725 (127.17.52.130:40493)
I20260927 03:21:42.227205 4294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.130@KRBTEST.COM'} at 127.17.52.130:36257
WARNING: no policy specified for kudu/127.17.52.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
Entry for principal kudu/127.17.52.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab.
I20260927 03:21:42.248919 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:42.332165 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.332404 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.332444 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.333575 4624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.333655 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.333702 4624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:42.335350 4624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.131.keytab
--principal=kudu/127.17.52.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.335637 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.335958 4624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.341127 4629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.341125 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.341121 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.341379 4624 server_base.cc:1034] running on GCE node
I20260927 03:21:42.341570 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.341802 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.342983 4624 hybrid_clock.cc:648] HybridClock initialized: now 1790479302342972 us; error 27 us; skew 500 ppm
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:42.345572 4624 init.cc:377] Logged in from keytab as kudu/127.17.52.131@KRBTEST.COM (short username kudu)
I20260927 03:21:42.345986 4624 webserver.cc:460] Webserver started at http://127.17.52.131:46617/ using document root <none> and password file <none>
I20260927 03:21:42.346220 4624 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.346287 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.346442 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.347436 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "af98036e149b4483b05d5e56c48900c9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "3e97a882bf32816a3556f8b92fbca6f5"
server_key_iv: "9665a926c7ff54e7f0307bf368165c01"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.347797 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "af98036e149b4483b05d5e56c48900c9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "3e97a882bf32816a3556f8b92fbca6f5"
server_key_iv: "9665a926c7ff54e7f0307bf368165c01"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.349201 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:21:42.350060 4639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.350286 4624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:42.350391 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "af98036e149b4483b05d5e56c48900c9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "3e97a882bf32816a3556f8b92fbca6f5"
server_key_iv: "9665a926c7ff54e7f0307bf368165c01"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.350478 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.366390 4624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.368369 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.368535 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.368794 4624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:42.369107 4624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:42.369163 4624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.369231 4624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:42.369261 4624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.375154 4624 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:42143
I20260927 03:21:42.375204 4752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:42143 every 8 connection(s)
I20260927 03:21:42.375519 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:21:42 dist-test-slave-7j43 krb5kdc[4249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.131@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:21:42.384225 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4624
I20260927 03:21:42.384357 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:42.384424 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.376239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:38255 (local address 127.17.52.190:45747)
0927 03:21:42.376325 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.376327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.376490 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:21:42.376992 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.376993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.377004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.377063 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.378064 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.378564 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.379533 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.379745 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.379802 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.381812 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.381813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.381814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:42.381822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.383091 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.383478 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.383479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.383479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.383537 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.383784 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.383785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.383785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.383894 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.383971 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.384253 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.384338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 03:21:42.384586 17618 external_mini_cluster.cc:1428] Setting key 14bd82a89518ab401f7cd29305968cdf
I20260927 03:21:42.384634 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.376204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:45747 (local address 127.17.52.131:38255)
0927 03:21:42.376356 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.376397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.376891 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.377116 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.377119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.377777 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:42.377968 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.377973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.378649 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.378650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.379450 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.379452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.379824 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.379825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.379925 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.380338 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:21:42.380346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:42.381713 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:42.383180 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.383186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.383188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.383398 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.383604 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:21:42.383605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:21:42.383605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:21:42.383714 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:21:42.383963 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:42.383969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:42.384108 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.384218 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.384370 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260927 03:21:42.385130 4753 heartbeater.cc:347] Connected to a master server at 127.17.52.190:45747
I20260927 03:21:42.385238 4753 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:42.385417 4753 heartbeater.cc:511] Master 127.17.52.190:45747 requested a full tablet report, sending...
I20260927 03:21:42.385813 4293 ts_manager.cc:194] Registered new tserver with Master: af98036e149b4483b05d5e56c48900c9 (127.17.52.131:42143)
I20260927 03:21:42.386204 4293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.52.131@KRBTEST.COM'} at 127.17.52.131:38255
I20260927 03:21:42.396456 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:42.403779 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.397952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.17.52.190:45747)
0927 03:21:42.398004 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.398005 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.398102 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:21:42.398147 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.398147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.398155 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.398211 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.398822 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.399312 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.400130 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.400315 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.400370 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.401498 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.401500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.401500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:42.401506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.402781 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.403089 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.403090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.403090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.403146 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:42.403328 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:42.403329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:42.403329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:42.403470 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.403557 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.403635 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.403715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:21:42.405541 4293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:42.407181 4293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:42.413519 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:42143 (local address 127.0.0.1:55176)
0927 03:21:42.409579 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.409616 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.409680 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.410032 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.410035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.410037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:42.410104 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.410106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411183 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.411184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.412346 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.412348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.413236 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.413237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.413313 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.413325 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.413391 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.413446 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":43,"threads_started":1}
I20260927 03:21:42.413532 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.17.52.131:42143)
0927 03:21:42.409838 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.409843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.409854 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:21:42.409899 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.409902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.409919 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.410011 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:42.410169 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411082 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.412432 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.413142 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.413234 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.413292 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.413419 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.413454 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260927 03:21:42.413763 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40343 (local address 127.0.0.1:34080)
0927 03:21:42.409784 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.409793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.409828 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.410282 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.410282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.410284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:42.410328 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.410329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411629 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.411630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.412672 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.412675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.413546 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.413547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.413611 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.413617 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.413652 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.413698 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":76,"threads_started":1}
I20260927 03:21:42.414147 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.17.52.129:40343)
0927 03:21:42.410093 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.410098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.410107 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:42.410145 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.410147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.410164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.410267 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:42.410411 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411516 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.412766 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.413443 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.413932 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.413993 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.414018 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.414055 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":109,"threads_started":1}
I20260927 03:21:42.414213 4687 tablet_service.cc:1467] Processing CreateTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 (DEFAULT_TABLE table=test-table [id=2a9b5b73edc744dcb0f8b03e9028d821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:42.414613 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebaf0ca92e2405aabc7ea898e30f2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:42.414760 4419 tablet_service.cc:1467] Processing CreateTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 (DEFAULT_TABLE table=test-table [id=2a9b5b73edc744dcb0f8b03e9028d821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:42.414943 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.17.52.130:40493)
0927 03:21:42.410069 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.410074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.410417 (+ 343us) server_negotiation.cc:407] Connection header received
0927 03:21:42.410486 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.410488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.410503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.410578 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:42.410696 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411702 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.413008 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.414083 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.414155 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.414206 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.414744 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.414794 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260927 03:21:42.414994 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebaf0ca92e2405aabc7ea898e30f2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:42.414630 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.409462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:40493 (local address 127.0.0.1:52448)
0927 03:21:42.410358 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.410367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.410421 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.410591 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.410592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.410593 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:42.410639 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.410640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.411797 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.411799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.412920 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.412923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.414166 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.414167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.414378 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.414383 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.414514 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.414559 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":85,"threads_started":1}
I20260927 03:21:42.417233 4553 tablet_service.cc:1467] Processing CreateTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 (DEFAULT_TABLE table=test-table [id=2a9b5b73edc744dcb0f8b03e9028d821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:42.417557 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebaf0ca92e2405aabc7ea898e30f2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:42.417546 4770 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Bootstrap starting.
I20260927 03:21:42.418147 4770 tablet_bootstrap.cc:654] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:42.418622 4770 log.cc:824] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:42.419256 4770 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: No bootstrap required, opened a new log
I20260927 03:21:42.419307 4770 ts_tablet_manager.cc:1397] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:42.420082 4773 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Bootstrap starting.
I20260927 03:21:42.420451 4771 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: Bootstrap starting.
I20260927 03:21:42.420818 4773 tablet_bootstrap.cc:654] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:42.420955 4770 raft_consensus.cc:348] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.421119 4770 raft_consensus.cc:374] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:42.421128 4771 tablet_bootstrap.cc:654] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:42.421161 4770 raft_consensus.cc:729] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af98036e149b4483b05d5e56c48900c9, State: Initialized, Role: FOLLOWER
I20260927 03:21:42.421289 4770 consensus_queue.cc:260] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.421444 4773 log.cc:824] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:42.421561 4770 ts_tablet_manager.cc:1428] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:42.421687 4753 heartbeater.cc:503] Master 127.17.52.190:45747 was elected leader, sending a full tablet report...
I20260927 03:21:42.421716 4771 log.cc:824] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:42.422477 4771 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: No bootstrap required, opened a new log
I20260927 03:21:42.422508 4774 raft_consensus.cc:482] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:42.422549 4771 ts_tablet_manager.cc:1397] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:42.422570 4774 raft_consensus.cc:504] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.422616 4773 tablet_bootstrap.cc:492] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: No bootstrap required, opened a new log
I20260927 03:21:42.422675 4773 ts_tablet_manager.cc:1397] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:21:42.423113 4774 leader_election.cc:288] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be134bcdefd844bda5d47917e2263d39 (127.17.52.129:40343), fee568e9fa1e4fe084e8129f64edf725 (127.17.52.130:40493)
I20260927 03:21:42.424387 4771 raft_consensus.cc:348] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.424537 4771 raft_consensus.cc:374] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:42.424569 4771 raft_consensus.cc:729] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be134bcdefd844bda5d47917e2263d39, State: Initialized, Role: FOLLOWER
I20260927 03:21:42.424695 4771 consensus_queue.cc:260] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.424934 4771 ts_tablet_manager.cc:1428] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:21:42.425276 4485 heartbeater.cc:503] Master 127.17.52.190:45747 was elected leader, sending a full tablet report...
I20260927 03:21:42.426193 4773 raft_consensus.cc:348] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.426398 4773 raft_consensus.cc:374] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:42.426452 4773 raft_consensus.cc:729] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee568e9fa1e4fe084e8129f64edf725, State: Initialized, Role: FOLLOWER
I20260927 03:21:42.426560 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.423001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40343 (local address 127.17.52.131:51993)
0927 03:21:42.423054 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.423066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.423129 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.423392 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.423393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.423395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:42.423456 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.423458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.424382 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.424383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.425551 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.425554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.426300 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.426301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.426390 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.426394 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.426454 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.426495 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:21:42.426560 4773 consensus_queue.cc:260] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.426745 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.423210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:51993 (local address 127.17.52.129:40343)
0927 03:21:42.423273 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.423274 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.423283 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:21:42.423303 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.423304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.423315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.423370 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:42.423537 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.424295 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.425643 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.426211 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.426281 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.426289 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.426642 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.426687 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:21:42.426807 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.423454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:41461 (local address 127.17.52.130:40493)
0927 03:21:42.423487 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.423489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.423495 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:21:42.423514 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.423515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.423524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.423577 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:42.423728 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.424387 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.425886 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.426444 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.426506 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.426514 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.426709 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.426750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:21:42.426826 4773 ts_tablet_manager.cc:1428] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:21:42.426983 4619 heartbeater.cc:503] Master 127.17.52.190:45747 was elected leader, sending a full tablet report...
I20260927 03:21:42.426968 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebaf0ca92e2405aabc7ea898e30f2a4" candidate_uuid: "af98036e149b4483b05d5e56c48900c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be134bcdefd844bda5d47917e2263d39" is_pre_election: true
I20260927 03:21:42.427011 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.423257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:40493 (local address 127.17.52.131:41461)
0927 03:21:42.423385 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.423394 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.423437 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.423594 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.423594 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.423596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:42.423646 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.423647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.424668 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.424669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.425821 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.425824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.426547 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.426548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.426619 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.426624 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.426897 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.426944 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 03:21:42.427071 4439 raft_consensus.cc:2454] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af98036e149b4483b05d5e56c48900c9 in term 0.
I20260927 03:21:42.427276 4641 leader_election.cc:302] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af98036e149b4483b05d5e56c48900c9, be134bcdefd844bda5d47917e2263d39; no voters:
I20260927 03:21:42.427496 4774 raft_consensus.cc:2781] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:42.427500 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebaf0ca92e2405aabc7ea898e30f2a4" candidate_uuid: "af98036e149b4483b05d5e56c48900c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee568e9fa1e4fe084e8129f64edf725" is_pre_election: true
I20260927 03:21:42.427605 4573 raft_consensus.cc:2454] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af98036e149b4483b05d5e56c48900c9 in term 0.
I20260927 03:21:42.427625 4774 raft_consensus.cc:482] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:42.427737 4774 raft_consensus.cc:3037] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:42.428346 4774 raft_consensus.cc:504] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.428577 4774 leader_election.cc:288] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [CANDIDATE]: Term 1 election: Requested vote from peers be134bcdefd844bda5d47917e2263d39 (127.17.52.129:40343), fee568e9fa1e4fe084e8129f64edf725 (127.17.52.130:40493)
I20260927 03:21:42.428611 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebaf0ca92e2405aabc7ea898e30f2a4" candidate_uuid: "af98036e149b4483b05d5e56c48900c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be134bcdefd844bda5d47917e2263d39"
I20260927 03:21:42.428678 4439 raft_consensus.cc:3037] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:42.429273 4439 raft_consensus.cc:2454] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af98036e149b4483b05d5e56c48900c9 in term 1.
I20260927 03:21:42.429409 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebaf0ca92e2405aabc7ea898e30f2a4" candidate_uuid: "af98036e149b4483b05d5e56c48900c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee568e9fa1e4fe084e8129f64edf725"
I20260927 03:21:42.429476 4641 leader_election.cc:302] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af98036e149b4483b05d5e56c48900c9, be134bcdefd844bda5d47917e2263d39; no voters:
I20260927 03:21:42.429481 4573 raft_consensus.cc:3037] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:42.429545 4774 raft_consensus.cc:2781] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:42.429673 4774 raft_consensus.cc:686] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 1 LEADER]: Becoming Leader. State: Replica: af98036e149b4483b05d5e56c48900c9, State: Running, Role: LEADER
I20260927 03:21:42.429754 4774 consensus_queue.cc:237] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } }
I20260927 03:21:42.430248 4573 raft_consensus.cc:2454] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af98036e149b4483b05d5e56c48900c9 in term 1.
I20260927 03:21:42.430456 4293 catalog_manager.cc:5495] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to af98036e149b4483b05d5e56c48900c9 (127.17.52.131). New cstate: current_term: 1 leader_uuid: "af98036e149b4483b05d5e56c48900c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af98036e149b4483b05d5e56c48900c9" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 42143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260927 03:21:42.437744 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.435599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.17.52.131:42143)
0927 03:21:42.435643 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.435645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.435652 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:21:42.435722 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.435722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.435736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.435784 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:42.435886 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.436338 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.437096 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.437272 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.437324 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.437434 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:42.437512 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:42.437562 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.437597 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.437611 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:21:42.439174 4439 raft_consensus.cc:1261] T eebaf0ca92e2405aabc7ea898e30f2a4 P be134bcdefd844bda5d47917e2263d39 [term 1 FOLLOWER]: Refusing update from remote peer af98036e149b4483b05d5e56c48900c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:42.439173 4573 raft_consensus.cc:1261] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 1 FOLLOWER]: Refusing update from remote peer af98036e149b4483b05d5e56c48900c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:42.439461 4774 consensus_queue.cc:1035] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee568e9fa1e4fe084e8129f64edf725" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:42.439610 4780 consensus_queue.cc:1035] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be134bcdefd844bda5d47917e2263d39" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:42.442006 4785 mvcc.cc:205] Tried to move back new op lower bound from 7333803222788853760 to 7333803222752993280. Current Snapshot: MvccSnapshot[applied={T|T < 7333803222788853760}]
I20260927 03:21:42.445379 4294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:42.445492 4294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:42.446303 4294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db57118535eb4cc7a0081a6d7dbd01ef: Sending DeleteTablet for 3 replicas of tablet eebaf0ca92e2405aabc7ea898e30f2a4
I20260927 03:21:42.446712 4687 tablet_service.cc:1514] Processing DeleteTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:42 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:55176
I20260927 03:21:42.446970 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:42 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:52448
I20260927 03:21:42.447029 4793 tablet_replica.cc:330] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: stopping tablet replica
I20260927 03:21:42.447116 4793 raft_consensus.cc:2229] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:21:42.447238 4793 raft_consensus.cc:2258] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:42.447450 4419 tablet_service.cc:1514] Processing DeleteTablet for tablet eebaf0ca92e2405aabc7ea898e30f2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:42 UTC) from {username='kudu', principal='kudu/127.17.52.190@KRBTEST.COM'} at 127.0.0.1:34080
I20260927 03:21:42.447644 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4356
I20260927 03:21:42.447680 4793 ts_tablet_manager.cc:1905] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:42.449445 4793 ts_tablet_manager.cc:1918] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:42.449519 4793 log.cc:1196] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/eebaf0ca92e2405aabc7ea898e30f2a4
I20260927 03:21:42.449792 4793 ts_tablet_manager.cc:1939] T eebaf0ca92e2405aabc7ea898e30f2a4 P af98036e149b4483b05d5e56c48900c9: Deleting consensus metadata
I20260927 03:21:42.450192 4281 catalog_manager.cc:4841] TS af98036e149b4483b05d5e56c48900c9 (127.17.52.131:42143): tablet eebaf0ca92e2405aabc7ea898e30f2a4 (table test-table [id=2a9b5b73edc744dcb0f8b03e9028d821]) successfully deleted
I20260927 03:21:42.450652 4794 tablet_replica.cc:330] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: stopping tablet replica
I20260927 03:21:42.450788 4794 raft_consensus.cc:2229] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:42.450893 4794 raft_consensus.cc:2258] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:42.451313 4794 ts_tablet_manager.cc:1905] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:42.453373 4794 ts_tablet_manager.cc:1918] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:42.453490 4794 log.cc:1196] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/eebaf0ca92e2405aabc7ea898e30f2a4
I20260927 03:21:42.453852 4794 ts_tablet_manager.cc:1939] T eebaf0ca92e2405aabc7ea898e30f2a4 P fee568e9fa1e4fe084e8129f64edf725: Deleting consensus metadata
I20260927 03:21:42.454345 4278 catalog_manager.cc:4841] TS fee568e9fa1e4fe084e8129f64edf725 (127.17.52.130:40493): tablet eebaf0ca92e2405aabc7ea898e30f2a4 (table test-table [id=2a9b5b73edc744dcb0f8b03e9028d821]) successfully deleted
W20260927 03:21:42.455027 4279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:40343 (error 108)
I20260927 03:21:42.455288 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4490
I20260927 03:21:42.455826 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.455381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:40343 (local address 127.0.0.1:34096)
0927 03:21:42.455757 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.455792 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
W20260927 03:21:42.456032 4279 catalog_manager.cc:4584] TS be134bcdefd844bda5d47917e2263d39 (127.17.52.129:40343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eebaf0ca92e2405aabc7ea898e30f2a4: Network error: Client connection negotiation failed: client connection to 127.17.52.129:40343: connect: Connection refused (error 111)
I20260927 03:21:42.460670 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4624
I20260927 03:21:42.465456 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4262
2026-09-27T03:21:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T03:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:42Z Disabled control of system clock
I20260927 03:21:42.493198 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40583
--webserver_interface=127.17.52.190
--webserver_port=0
--builtin_ntp_servers=127.17.52.148:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:42.572395 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.572607 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.572630 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.573673 4800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:42.573706 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.573719 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.573732 4800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:42.573745 4800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:42.574949 4800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:40583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.575119 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.575322 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.579840 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.579852 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.579852 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.580298 4800 server_base.cc:1034] running on GCE node
I20260927 03:21:42.580498 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.580798 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.581941 4800 hybrid_clock.cc:648] HybridClock initialized: now 1790479302581930 us; error 26 us; skew 500 ppm
I20260927 03:21:42.583098 4800 webserver.cc:460] Webserver started at http://127.17.52.190:45793/ using document root <none> and password file <none>
I20260927 03:21:42.583328 4800 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.583372 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.583532 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.584554 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "081c6767af6b43bdbdb54fe5bac56ea9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "e15c915ac6be5fe9ff340b5ede58bf47"
server_key_iv: "4ef2e72907c7dd58f4ec59961b1fdcdb"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.584983 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "081c6767af6b43bdbdb54fe5bac56ea9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "e15c915ac6be5fe9ff340b5ede58bf47"
server_key_iv: "4ef2e72907c7dd58f4ec59961b1fdcdb"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.586437 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:42.587172 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.587335 4800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.587435 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "081c6767af6b43bdbdb54fe5bac56ea9"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "e15c915ac6be5fe9ff340b5ede58bf47"
server_key_iv: "4ef2e72907c7dd58f4ec59961b1fdcdb"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.587505 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.598256 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.598526 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.598673 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.602591 4800 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:40583
I20260927 03:21:42.602623 4866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:40583 every 8 connection(s)
I20260927 03:21:42.602939 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:42.603816 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:42.606050 4867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: Bootstrap starting.
I20260927 03:21:42.606609 4867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:42.606902 4867 log.cc:824] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:42.607227 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4800
I20260927 03:21:42.607345 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:42.607558 17618 external_mini_cluster.cc:1428] Setting key cb76bb70ec9475c3d51e2174f472956d
I20260927 03:21:42.607738 4867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: No bootstrap required, opened a new log
I20260927 03:21:42.610570 4867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } }
I20260927 03:21:42.610729 4867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:42.610759 4867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081c6767af6b43bdbdb54fe5bac56ea9, State: Initialized, Role: FOLLOWER
I20260927 03:21:42.610877 4867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } }
I20260927 03:21:42.610946 4867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:42.610982 4867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:42.611038 4867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:42.611563 4867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } }
I20260927 03:21:42.611646 4867 leader_election.cc:302] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 081c6767af6b43bdbdb54fe5bac56ea9; no voters:
I20260927 03:21:42.611833 4867 leader_election.cc:288] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:42.611858 4872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:42.611974 4872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 081c6767af6b43bdbdb54fe5bac56ea9, State: Running, Role: LEADER
I20260927 03:21:42.612062 4872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } }
I20260927 03:21:42.612104 4867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:42.612376 4873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 081c6767af6b43bdbdb54fe5bac56ea9. Latest consensus state: current_term: 1 leader_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } } }
I20260927 03:21:42.612479 4873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:42.612689 4874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c6767af6b43bdbdb54fe5bac56ea9" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 40583 } } }
I20260927 03:21:42.612759 4874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:42.612879 4878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:42.613391 4878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:42.614380 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.608073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.17.52.190:40583)
0927 03:21:42.608292 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.608301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.608342 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:21:42.609122 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.609131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.609205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.609362 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.609529 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.610940 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.611839 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.612466 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.612949 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.613399 (+ 450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.613403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.613404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:42.613627 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.613677 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:42.613694 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.613758 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.614068 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.614149 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260927 03:21:42.615213 4878 catalog_manager.cc:1347] Generated new cluster ID: 97e68fb3f66b42048f26fa209d01e5f4
I20260927 03:21:42.615270 4878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:42.644346 4878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:42.644950 4878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:42.649880 4878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: Generated new TSK 0
I20260927 03:21:42.650050 4878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:42.662647 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.129:0
--local_ip_for_outbound_sockets=127.17.52.129
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--builtin_ntp_servers=127.17.52.148:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:42.743153 4891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.743336 4891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.743356 4891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.744494 4891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.744537 4891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.744561 4891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.129
I20260927 03:21:42.745914 4891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.52.129
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.129
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.746140 4891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.746424 4891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.751307 4897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.751338 4896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.751353 4899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.751380 4891 server_base.cc:1034] running on GCE node
I20260927 03:21:42.751633 4891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.751817 4891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.752985 4891 hybrid_clock.cc:648] HybridClock initialized: now 1790479302752972 us; error 24 us; skew 500 ppm
I20260927 03:21:42.754164 4891 webserver.cc:460] Webserver started at http://127.17.52.129:41843/ using document root <none> and password file <none>
I20260927 03:21:42.754417 4891 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.754490 4891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.754640 4891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.755766 4891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data/instance:
uuid: "2133a00d30b248f2bc176264d79b9a60"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "08ce9915f25080fb6ea9ddbc8f98f86a"
server_key_iv: "682fc09c34a24d10b03198296a664303"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.756141 4891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance:
uuid: "2133a00d30b248f2bc176264d79b9a60"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "08ce9915f25080fb6ea9ddbc8f98f86a"
server_key_iv: "682fc09c34a24d10b03198296a664303"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.757512 4891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:21:42.758226 4905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.758386 4891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:42.758484 4891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal
uuid: "2133a00d30b248f2bc176264d79b9a60"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "08ce9915f25080fb6ea9ddbc8f98f86a"
server_key_iv: "682fc09c34a24d10b03198296a664303"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.758569 4891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.770440 4891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.770714 4891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.770855 4891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.771060 4891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:42.771344 4891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:42.771400 4891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.771440 4891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:42.771467 4891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.777071 4891 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.129:46837
I20260927 03:21:42.777153 5018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.129:46837 every 8 connection(s)
I20260927 03:21:42.777437 4891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/data/info.pb
I20260927 03:21:42.782454 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.778145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.129:38623 (local address 127.17.52.190:40583)
0927 03:21:42.778213 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.778214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.778437 (+ 223us) server_negotiation.cc:407] Connection header received
0927 03:21:42.778955 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.778957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.778974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.779026 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.779650 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.780151 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.781146 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.781356 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.781438 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.782174 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.782176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.782176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:42.782180 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.782195 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:42.782199 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.782246 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.782355 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.782378 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:21:42.782667 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.778131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40583 (local address 127.17.52.129:38623)
0927 03:21:42.778320 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.778352 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.778837 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.779089 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.779093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.779102 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:42.779559 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.779563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.780244 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.780245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.781046 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.781048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.781447 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.781448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.781592 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.782039 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:42.782040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:42.782065 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:42.782084 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:42.782245 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:42.782250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:42.782253 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.782335 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.782432 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 03:21:42.783125 5019 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40583
I20260927 03:21:42.783224 5019 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:42.783353 5019 heartbeater.cc:511] Master 127.17.52.190:40583 requested a full tablet report, sending...
I20260927 03:21:42.783864 4831 ts_manager.cc:194] Registered new tserver with Master: 2133a00d30b248f2bc176264d79b9a60 (127.17.52.129:46837)
I20260927 03:21:42.784446 4831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.129:38623
I20260927 03:21:42.787318 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 4891
I20260927 03:21:42.787431 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-0/wal/instance
I20260927 03:21:42.787619 17618 external_mini_cluster.cc:1428] Setting key 22e4b33fd87aaad14483f796a5b2d240
I20260927 03:21:42.788496 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.130:0
--local_ip_for_outbound_sockets=127.17.52.130
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--builtin_ntp_servers=127.17.52.148:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:21:42.868117 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.868358 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.868392 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.869557 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.869627 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.869671 5022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.130
I20260927 03:21:42.871029 5022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.52.130
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.130
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.871289 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.871579 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:42.876477 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.876531 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:42.876483 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:42.876786 5022 server_base.cc:1034] running on GCE node
I20260927 03:21:42.876993 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:42.877182 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:42.878355 5022 hybrid_clock.cc:648] HybridClock initialized: now 1790479302878336 us; error 51 us; skew 500 ppm
I20260927 03:21:42.879725 5022 webserver.cc:460] Webserver started at http://127.17.52.130:42909/ using document root <none> and password file <none>
I20260927 03:21:42.879966 5022 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:42.880038 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:42.880187 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:42.881297 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data/instance:
uuid: "95766fb9f5fe41beb3a462306908e5ac"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9727d28df2fb80a260cc4cbeba4a4bcf"
server_key_iv: "b7c8296f5abbd303a898b03a8c370d5a"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.881647 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance:
uuid: "95766fb9f5fe41beb3a462306908e5ac"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9727d28df2fb80a260cc4cbeba4a4bcf"
server_key_iv: "b7c8296f5abbd303a898b03a8c370d5a"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.882949 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:21:42.883630 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.883807 5022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:42.883905 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal
uuid: "95766fb9f5fe41beb3a462306908e5ac"
format_stamp: "Formatted at 2026-09-27 03:21:42 on dist-test-slave-7j43"
server_key: "9727d28df2fb80a260cc4cbeba4a4bcf"
server_key_iv: "b7c8296f5abbd303a898b03a8c370d5a"
server_key_version: "encryptionkey@0"
I20260927 03:21:42.883983 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:42.900825 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:42.901134 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:42.901285 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:42.901511 5022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:42.901863 5022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:42.901919 5022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.901959 5022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:42.901988 5022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:42.907709 5022 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.130:35263
I20260927 03:21:42.907742 5149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.130:35263 every 8 connection(s)
I20260927 03:21:42.908075 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/data/info.pb
I20260927 03:21:42.913455 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5022
I20260927 03:21:42.913503 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.908889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:51677 (local address 127.17.52.190:40583)
0927 03:21:42.908975 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:42.908977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:42.909175 (+ 198us) server_negotiation.cc:407] Connection header received
0927 03:21:42.909704 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:42.909705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:42.909716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:42.909778 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:42.910483 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.910979 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.912010 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.912198 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.912253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.913165 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:42.913166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:42.913166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:42.913189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:42.913203 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:42.913207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:42.913297 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:42.913394 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 03:21:42.913422 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:21:42.913586 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal/instance
I20260927 03:21:42.913756 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:42.908852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40583 (local address 127.17.52.130:51677)
0927 03:21:42.909051 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:42.909090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:42.909581 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:42.909876 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:42.909879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:42.909888 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:42.910392 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.910396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.911066 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.911067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.911932 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:42.911934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:42.912293 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:42.912295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:42.912433 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:42.913006 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:42.913006 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:42.913036 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:42.913077 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:42.913300 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:42.913306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:42.913309 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:42.913397 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:21:42.913500 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260927 03:21:42.913790 17618 external_mini_cluster.cc:1428] Setting key bd0df8a7d8d1aa884ae66694906061e5
I20260927 03:21:42.914373 5150 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40583
I20260927 03:21:42.914464 5150 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:42.914685 5150 heartbeater.cc:511] Master 127.17.52.190:40583 requested a full tablet report, sending...
I20260927 03:21:42.914969 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.52.131:0
--local_ip_for_outbound_sockets=127.17.52.131
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--builtin_ntp_servers=127.17.52.148:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:21:42.915096 4830 ts_manager.cc:194] Registered new tserver with Master: 95766fb9f5fe41beb3a462306908e5ac (127.17.52.130:35263)
I20260927 03:21:42.915467 4830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.130:51677
W20260927 03:21:42.994969 5153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:42.995208 5153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:42.995230 5153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:42.996378 5153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:42.996418 5153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:42.996443 5153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.52.131
I20260927 03:21:42.997825 5153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.52.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.52.131
--webserver_port=0
--tserver_master_addrs=127.17.52.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.52.131
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:42.998070 5153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:42.998360 5153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:43.003649 5161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:43.003741 5153 server_base.cc:1034] running on GCE node
W20260927 03:21:43.003631 5159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:43.003628 5158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:43.004030 5153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:43.004262 5153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:43.005430 5153 hybrid_clock.cc:648] HybridClock initialized: now 1790479303005417 us; error 43 us; skew 500 ppm
I20260927 03:21:43.006886 5153 webserver.cc:460] Webserver started at http://127.17.52.131:39619/ using document root <none> and password file <none>
I20260927 03:21:43.007138 5153 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:43.007189 5153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:43.007381 5153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:43.008468 5153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data/instance:
uuid: "205e3b321ab24dcf8b70e3399e212515"
format_stamp: "Formatted at 2026-09-27 03:21:43 on dist-test-slave-7j43"
server_key: "07d15b8c35e7a832188745c6a36bcab5"
server_key_iv: "38097560674686c533291120366a387a"
server_key_version: "encryptionkey@0"
I20260927 03:21:43.008909 5153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance:
uuid: "205e3b321ab24dcf8b70e3399e212515"
format_stamp: "Formatted at 2026-09-27 03:21:43 on dist-test-slave-7j43"
server_key: "07d15b8c35e7a832188745c6a36bcab5"
server_key_iv: "38097560674686c533291120366a387a"
server_key_version: "encryptionkey@0"
I20260927 03:21:43.010213 5153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:43.010927 5167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:43.011103 5153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:43.011206 5153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal
uuid: "205e3b321ab24dcf8b70e3399e212515"
format_stamp: "Formatted at 2026-09-27 03:21:43 on dist-test-slave-7j43"
server_key: "07d15b8c35e7a832188745c6a36bcab5"
server_key_iv: "38097560674686c533291120366a387a"
server_key_version: "encryptionkey@0"
I20260927 03:21:43.011303 5153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:43.029713 5153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:43.030057 5153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:43.030231 5153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:43.030479 5153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:21:43.030829 5153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:21:43.030886 5153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:43.030930 5153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:21:43.030961 5153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:43.036901 5153 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.131:37295
I20260927 03:21:43.036917 5280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.131:37295 every 8 connection(s)
I20260927 03:21:43.037374 5153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/data/info.pb
I20260927 03:21:43.039907 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5153
I20260927 03:21:43.040024 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal/instance
I20260927 03:21:43.040210 17618 external_mini_cluster.cc:1428] Setting key 2dfb71a61fcd821832ad6fec8941e09f
I20260927 03:21:43.043144 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.038169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.131:35569 (local address 127.17.52.190:40583)
0927 03:21:43.038233 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.038235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.038457 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:21:43.038934 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.038935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.038945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.039000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:43.039619 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.040218 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.041683 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.041904 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.041982 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.042872 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:43.042873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:43.042874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:43.042879 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:43.042892 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:43.042896 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:43.042945 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.043038 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.043065 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:21:43.043371 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.038161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:40583 (local address 127.17.52.131:35569)
0927 03:21:43.038331 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.038363 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.038837 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.039054 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.039057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.039065 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:21:43.039535 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.039538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.040328 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.040329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.041586 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.041588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.041999 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.042001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.042140 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.042717 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:21:43.042718 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:21:43.042748 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:21:43.042771 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:21:43.042945 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:21:43.042950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:21:43.042953 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.043024 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.043128 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260927 03:21:43.043840 5281 heartbeater.cc:347] Connected to a master server at 127.17.52.190:40583
I20260927 03:21:43.043921 5281 heartbeater.cc:464] Registering TS with master...
I20260927 03:21:43.044061 5281 heartbeater.cc:511] Master 127.17.52.190:40583 requested a full tablet report, sending...
I20260927 03:21:43.044376 4830 ts_manager.cc:194] Registered new tserver with Master: 205e3b321ab24dcf8b70e3399e212515 (127.17.52.131:37295)
I20260927 03:21:43.044802 4830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.52.131:35569
I20260927 03:21:43.052279 17618 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:21:43.056610 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.053677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.17.52.190:40583)
0927 03:21:43.053752 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.053754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.053846 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:21:43.053917 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.053918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.053928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.053979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:43.054159 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.054642 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.055382 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.055576 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.055659 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.056344 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:43.056345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:43.056346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:43.056350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:43.056364 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:43.056368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:43.056432 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.056484 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.056506 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:21:43.058231 4830 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:21:43.059782 4830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:21:43.066486 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.061885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.130:35263 (local address 127.0.0.1:42652)
0927 03:21:43.062202 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.062247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.062320 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.062645 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.062648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.062651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:43.062727 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.062729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.064000 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.064001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.065283 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.065286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.066186 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.066187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.066262 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.066277 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.066363 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.066412 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":51,"threads_started":1}
I20260927 03:21:43.066499 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.062100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.17.52.130:35263)
0927 03:21:43.062471 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.062476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.062486 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:21:43.062526 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.062528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.062544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.062627 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:43.062798 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.063925 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.065381 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.066107 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.066173 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.066239 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.066386 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.066422 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":73,"threads_started":1}
I20260927 03:21:43.067224 5084 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d (DEFAULT_TABLE table=test-table [id=0a4cf8d98faf45d58a575b4ad0a05c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:43.067430 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.062103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:37295 (local address 127.0.0.1:49596)
0927 03:21:43.062729 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.062738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.062787 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.062975 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.062975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.062977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:43.063014 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.063015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.064169 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.064170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.065283 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.065286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.067174 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.067175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.067242 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.067247 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.067300 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.067347 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260927 03:21:43.067430 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.062186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.17.52.131:37295)
0927 03:21:43.062355 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.062359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.062848 (+ 489us) server_negotiation.cc:407] Connection header received
0927 03:21:43.062891 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.062894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.062908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.062981 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:43.063075 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.064087 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.066270 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.066980 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.067052 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.067142 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.067313 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.067349 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260927 03:21:43.067632 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea33fb8961744719aabaaf4ff7aab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:43.067754 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.061905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46837 (local address 127.0.0.1:45350)
0927 03:21:43.062265 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.062274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.062320 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.062672 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.062672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.062674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:43.062728 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.062729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.065001 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.065002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.066003 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.066006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.067226 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.067227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.067290 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.067295 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.067325 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.067366 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":18,"thread_start_us":123,"threads_started":1}
I20260927 03:21:43.068120 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.062013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.17.52.129:46837)
0927 03:21:43.062471 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.062476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.062486 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:21:43.062526 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.062528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.062568 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.062667 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:43.064025 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.064908 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.066089 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.067125 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.067876 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.067966 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.067991 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.068033 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":79,"threads_started":1}
I20260927 03:21:43.068095 5215 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d (DEFAULT_TABLE table=test-table [id=0a4cf8d98faf45d58a575b4ad0a05c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:43.068425 5215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea33fb8961744719aabaaf4ff7aab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:43.068766 4953 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d (DEFAULT_TABLE table=test-table [id=0a4cf8d98faf45d58a575b4ad0a05c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:21:43.069090 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea33fb8961744719aabaaf4ff7aab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:43.070560 5295 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Bootstrap starting.
I20260927 03:21:43.071091 5295 tablet_bootstrap.cc:654] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:43.071386 5295 log.cc:824] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:43.072007 5297 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Bootstrap starting.
I20260927 03:21:43.072052 5295 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: No bootstrap required, opened a new log
I20260927 03:21:43.072124 5295 ts_tablet_manager.cc:1397] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:43.072407 5298 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: Bootstrap starting.
I20260927 03:21:43.072707 5297 tablet_bootstrap.cc:654] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:43.073071 5297 log.cc:824] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:43.073273 5298 tablet_bootstrap.cc:654] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:43.073611 5298 log.cc:824] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:43.073803 5297 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: No bootstrap required, opened a new log
I20260927 03:21:43.073874 5297 ts_tablet_manager.cc:1397] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:43.074457 5298 tablet_bootstrap.cc:492] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: No bootstrap required, opened a new log
I20260927 03:21:43.074527 5298 ts_tablet_manager.cc:1397] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:21:43.074800 5295 raft_consensus.cc:348] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.074998 5295 raft_consensus.cc:374] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:43.075048 5295 raft_consensus.cc:729] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95766fb9f5fe41beb3a462306908e5ac, State: Initialized, Role: FOLLOWER
I20260927 03:21:43.075174 5295 consensus_queue.cc:260] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.075435 5295 ts_tablet_manager.cc:1428] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:21:43.075553 5150 heartbeater.cc:503] Master 127.17.52.190:40583 was elected leader, sending a full tablet report...
I20260927 03:21:43.076251 5297 raft_consensus.cc:348] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.076411 5297 raft_consensus.cc:374] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:43.076452 5297 raft_consensus.cc:729] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205e3b321ab24dcf8b70e3399e212515, State: Initialized, Role: FOLLOWER
I20260927 03:21:43.076591 5297 consensus_queue.cc:260] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.076879 5281 heartbeater.cc:503] Master 127.17.52.190:40583 was elected leader, sending a full tablet report...
I20260927 03:21:43.077126 5297 ts_tablet_manager.cc:1428] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:43.077075 5298 raft_consensus.cc:348] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.077235 5298 raft_consensus.cc:374] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:43.077298 5298 raft_consensus.cc:729] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2133a00d30b248f2bc176264d79b9a60, State: Initialized, Role: FOLLOWER
I20260927 03:21:43.077416 5298 consensus_queue.cc:260] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.077638 5298 ts_tablet_manager.cc:1428] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:21:43.077704 5019 heartbeater.cc:503] Master 127.17.52.190:40583 was elected leader, sending a full tablet report...
I20260927 03:21:43.141696 5301 raft_consensus.cc:482] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:21:43.141817 5301 raft_consensus.cc:504] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.142138 5301 leader_election.cc:288] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 205e3b321ab24dcf8b70e3399e212515 (127.17.52.131:37295), 2133a00d30b248f2bc176264d79b9a60 (127.17.52.129:46837)
I20260927 03:21:43.145884 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.142491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:55579 (local address 127.17.52.129:46837)
0927 03:21:43.142564 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.142566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.142574 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:21:43.142606 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.142607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.142619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.142675 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:43.142831 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.143515 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.144954 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.145501 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.145574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.145582 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.145784 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.145823 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:21:43.145884 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.142382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.52.130:55905 (local address 127.17.52.131:37295)
0927 03:21:43.142452 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.142454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.142462 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:21:43.142488 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.142489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.142496 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.142582 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:21:43.142741 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.143507 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.144954 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.145498 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.145574 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.145582 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.145760 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.145801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:21:43.145886 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.142307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.131:37295 (local address 127.17.52.130:55905)
0927 03:21:43.142364 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.142377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.142419 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.142584 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.142585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.142587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:43.142660 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.142664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.143593 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.143593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.144863 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.144866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.145591 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.145592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.145668 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.145673 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.145731 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.145784 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":12}
I20260927 03:21:43.145892 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.142320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46837 (local address 127.17.52.130:55579)
0927 03:21:43.142515 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.142523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:21:43.142561 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:21:43.142691 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:21:43.142692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:21:43.142693 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:21:43.142758 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.142759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.143593 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.143593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.144865 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:21:43.144866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.145592 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.145593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:21:43.145679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.145682 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:21:43.145767 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:21:43.145794 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260927 03:21:43.146443 4973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea33fb8961744719aabaaf4ff7aab1d" candidate_uuid: "95766fb9f5fe41beb3a462306908e5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2133a00d30b248f2bc176264d79b9a60" is_pre_election: true
I20260927 03:21:43.146538 5235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea33fb8961744719aabaaf4ff7aab1d" candidate_uuid: "95766fb9f5fe41beb3a462306908e5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205e3b321ab24dcf8b70e3399e212515" is_pre_election: true
I20260927 03:21:43.146617 4973 raft_consensus.cc:2454] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95766fb9f5fe41beb3a462306908e5ac in term 0.
I20260927 03:21:43.146698 5235 raft_consensus.cc:2454] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95766fb9f5fe41beb3a462306908e5ac in term 0.
I20260927 03:21:43.146860 5037 leader_election.cc:302] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2133a00d30b248f2bc176264d79b9a60, 95766fb9f5fe41beb3a462306908e5ac; no voters:
I20260927 03:21:43.146987 5301 raft_consensus.cc:2781] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:21:43.147051 5301 raft_consensus.cc:482] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:21:43.147087 5301 raft_consensus.cc:3037] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:43.147766 5301 raft_consensus.cc:504] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.147908 5301 leader_election.cc:288] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [CANDIDATE]: Term 1 election: Requested vote from peers 205e3b321ab24dcf8b70e3399e212515 (127.17.52.131:37295), 2133a00d30b248f2bc176264d79b9a60 (127.17.52.129:46837)
I20260927 03:21:43.148049 4973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea33fb8961744719aabaaf4ff7aab1d" candidate_uuid: "95766fb9f5fe41beb3a462306908e5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2133a00d30b248f2bc176264d79b9a60"
I20260927 03:21:43.148041 5235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea33fb8961744719aabaaf4ff7aab1d" candidate_uuid: "95766fb9f5fe41beb3a462306908e5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205e3b321ab24dcf8b70e3399e212515"
I20260927 03:21:43.148109 4973 raft_consensus.cc:3037] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:43.148109 5235 raft_consensus.cc:3037] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:43.149024 4973 raft_consensus.cc:2454] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95766fb9f5fe41beb3a462306908e5ac in term 1.
I20260927 03:21:43.149024 5235 raft_consensus.cc:2454] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95766fb9f5fe41beb3a462306908e5ac in term 1.
I20260927 03:21:43.149232 5039 leader_election.cc:302] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205e3b321ab24dcf8b70e3399e212515, 95766fb9f5fe41beb3a462306908e5ac; no voters:
I20260927 03:21:43.149341 5301 raft_consensus.cc:2781] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:43.149520 5301 raft_consensus.cc:686] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 1 LEADER]: Becoming Leader. State: Replica: 95766fb9f5fe41beb3a462306908e5ac, State: Running, Role: LEADER
I20260927 03:21:43.149616 5301 consensus_queue.cc:237] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } }
I20260927 03:21:43.150354 4830 catalog_manager.cc:5495] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 95766fb9f5fe41beb3a462306908e5ac (127.17.52.130). New cstate: current_term: 1 leader_uuid: "95766fb9f5fe41beb3a462306908e5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95766fb9f5fe41beb3a462306908e5ac" member_type: VOTER last_known_addr { host: "127.17.52.130" port: 35263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 } health_report { overall_health: UNKNOWN } } }
W20260927 03:21:43.158579 5151 tablet.cc:2369] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:21:43.159529 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.157374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.17.52.130:35263)
0927 03:21:43.157418 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:43.157420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:43.157427 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:21:43.157459 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:43.157460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:43.157473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:43.157533 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:21:43.157609 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.158094 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.158875 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:43.159053 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:43.159106 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:43.159232 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:21:43.159319 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:21:43.159373 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:43.159407 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:21:43.159421 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:21:43.161227 4973 raft_consensus.cc:1261] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 1 FOLLOWER]: Refusing update from remote peer 95766fb9f5fe41beb3a462306908e5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:43.161247 5235 raft_consensus.cc:1261] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 1 FOLLOWER]: Refusing update from remote peer 95766fb9f5fe41beb3a462306908e5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:21:43.161561 5305 consensus_queue.cc:1035] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "205e3b321ab24dcf8b70e3399e212515" member_type: VOTER last_known_addr { host: "127.17.52.131" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:43.161687 5301 consensus_queue.cc:1035] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2133a00d30b248f2bc176264d79b9a60" member_type: VOTER last_known_addr { host: "127.17.52.129" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:21:43.164624 5309 mvcc.cc:205] Tried to move back new op lower bound from 7333803225745420288 to 7333803225701347328. Current Snapshot: MvccSnapshot[applied={T|T < 7333803225745420288}]
I20260927 03:21:43.166270 4830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:43.166368 4830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:21:43.167227 4830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 081c6767af6b43bdbdb54fe5bac56ea9: Sending DeleteTablet for 3 replicas of tablet 9ea33fb8961744719aabaaf4ff7aab1d
I20260927 03:21:43.167608 5215 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:43 UTC) from {username='slave'} at 127.0.0.1:49596
I20260927 03:21:43.167685 4953 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:43 UTC) from {username='slave'} at 127.0.0.1:45350
I20260927 03:21:43.167873 5318 tablet_replica.cc:330] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60: stopping tablet replica
I20260927 03:21:43.167898 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4891
I20260927 03:21:43.167968 5318 raft_consensus.cc:2229] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:43.168061 5318 raft_consensus.cc:2258] T 9ea33fb8961744719aabaaf4ff7aab1d P 2133a00d30b248f2bc176264d79b9a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:43.171768 5084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ea33fb8961744719aabaaf4ff7aab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:21:43 UTC) from {username='slave'} at 127.0.0.1:42652
I20260927 03:21:43.171988 5319 tablet_replica.cc:330] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: stopping tablet replica
I20260927 03:21:43.172107 5319 raft_consensus.cc:2229] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:21:43.172211 5319 raft_consensus.cc:2258] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:43.172374 5317 tablet_replica.cc:330] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: stopping tablet replica
I20260927 03:21:43.172465 5317 raft_consensus.cc:2229] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:21:43.172588 5317 raft_consensus.cc:2258] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:21:43.172785 5319 ts_tablet_manager.cc:1905] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:43.173017 5317 ts_tablet_manager.cc:1905] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:21:43.174444 5319 ts_tablet_manager.cc:1918] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:43.174512 5319 log.cc:1196] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-1/wal/wals/9ea33fb8961744719aabaaf4ff7aab1d
I20260927 03:21:43.174778 5319 ts_tablet_manager.cc:1939] T 9ea33fb8961744719aabaaf4ff7aab1d P 95766fb9f5fe41beb3a462306908e5ac: Deleting consensus metadata
I20260927 03:21:43.175172 4817 catalog_manager.cc:4841] TS 95766fb9f5fe41beb3a462306908e5ac (127.17.52.130:35263): tablet 9ea33fb8961744719aabaaf4ff7aab1d (table test-table [id=0a4cf8d98faf45d58a575b4ad0a05c46]) successfully deleted
I20260927 03:21:43.175350 5317 ts_tablet_manager.cc:1918] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:21:43.175441 5317 log.cc:1196] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Deleting WAL directory at /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479066103559-17618-0/minicluster-data/ts-2/wal/wals/9ea33fb8961744719aabaaf4ff7aab1d
W20260927 03:21:43.175689 4815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.52.129:46837 (error 108)
I20260927 03:21:43.175813 5317 ts_tablet_manager.cc:1939] T 9ea33fb8961744719aabaaf4ff7aab1d P 205e3b321ab24dcf8b70e3399e212515: Deleting consensus metadata
I20260927 03:21:43.176185 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:43.176045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.129:46837 (local address 127.0.0.1:45354)
0927 03:21:43.176106 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:21:43.176153 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:21:43.176203 4817 catalog_manager.cc:4841] TS 205e3b321ab24dcf8b70e3399e212515 (127.17.52.131:37295): tablet 9ea33fb8961744719aabaaf4ff7aab1d (table test-table [id=0a4cf8d98faf45d58a575b4ad0a05c46]) successfully deleted
W20260927 03:21:43.176333 4815 catalog_manager.cc:4584] TS 2133a00d30b248f2bc176264d79b9a60 (127.17.52.129:46837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ea33fb8961744719aabaaf4ff7aab1d: Network error: Client connection negotiation failed: client connection to 127.17.52.129:46837: connect: Connection refused (error 111)
I20260927 03:21:43.176661 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5022
I20260927 03:21:43.180696 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5153
I20260927 03:21:43.184625 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 4800
2026-09-27T03:21:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (718 ms)
[----------] 6 tests from EncryptionPolicyTest (8174 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:43 dist-test-slave-7j43 krb5kdc[5323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:43 dist-test-slave-7j43 krb5kdc[5323](info): set up 2 sockets
Sep 27 03:21:43 dist-test-slave-7j43 krb5kdc[5323](info): commencing operation
krb5kdc: starting...
W20260927 03:21:44.229420 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:44.351251 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:44.431411 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:44.431614 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:44.431638 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:44.432837 5336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:44.432883 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:44.432896 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:44.432909 5336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:44.432969 5336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:44.434443 5336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:44.434681 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:44.434937 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:44.439689 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:44.439680 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:44.439680 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:44.439983 5336 server_base.cc:1034] running on GCE node
I20260927 03:21:44.440199 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:44.440450 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:44.441592 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790479304441582 us; error 51 us; skew 500 ppm
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:44.444106 5336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:44.444640 5336 webserver.cc:460] Webserver started at http://127.0.0.1:44621/ using document root <none> and password file <none>
I20260927 03:21:44.444888 5336 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:44.444936 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:44.445079 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:44.446144 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "53c96a6531884e60977f59458e31dd99"
format_stamp: "Formatted at 2026-09-27 03:21:44 on dist-test-slave-7j43"
server_key: "0ca54915902d653214f14983abcaf9ed"
server_key_iv: "6fb1fec98fd597e19ab2992f3d7fbc05"
server_key_version: "encryptionkey@0"
I20260927 03:21:44.446521 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "53c96a6531884e60977f59458e31dd99"
format_stamp: "Formatted at 2026-09-27 03:21:44 on dist-test-slave-7j43"
server_key: "0ca54915902d653214f14983abcaf9ed"
server_key_iv: "6fb1fec98fd597e19ab2992f3d7fbc05"
server_key_version: "encryptionkey@0"
I20260927 03:21:44.447878 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:44.448719 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:44.448899 5336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:44.448998 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "53c96a6531884e60977f59458e31dd99"
format_stamp: "Formatted at 2026-09-27 03:21:44 on dist-test-slave-7j43"
server_key: "0ca54915902d653214f14983abcaf9ed"
server_key_iv: "6fb1fec98fd597e19ab2992f3d7fbc05"
server_key_version: "encryptionkey@0"
I20260927 03:21:44.449080 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:44.465785 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:44.468082 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:44.468241 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:44.472422 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35821
I20260927 03:21:44.472463 5403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35821 every 8 connection(s)
I20260927 03:21:44.472859 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:44.473708 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:44.475921 5404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99: Bootstrap starting.
I20260927 03:21:44.476161 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5336
I20260927 03:21:44.476295 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:44.476495 17618 external_mini_cluster.cc:1428] Setting key 268f633fba074f183edb63a981e0d3c7
I20260927 03:21:44.476626 5404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:44.476939 5404 log.cc:824] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:44.477751 5404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99: No bootstrap required, opened a new log
I20260927 03:21:44.480135 5404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } }
I20260927 03:21:44.480291 5404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:44.480319 5404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c96a6531884e60977f59458e31dd99, State: Initialized, Role: FOLLOWER
I20260927 03:21:44.480443 5404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } }
I20260927 03:21:44.480530 5404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:44.480616 5404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:44.480671 5404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:44.481187 5404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } }
I20260927 03:21:44.481310 5404 leader_election.cc:302] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c96a6531884e60977f59458e31dd99; no voters:
I20260927 03:21:44.481473 5404 leader_election.cc:288] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:44.481535 5409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:44.481694 5404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:44.481714 5409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [term 1 LEADER]: Becoming Leader. State: Replica: 53c96a6531884e60977f59458e31dd99, State: Running, Role: LEADER
I20260927 03:21:44.481797 5409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } }
I20260927 03:21:44.482074 5409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c96a6531884e60977f59458e31dd99. Latest consensus state: current_term: 1 leader_uuid: "53c96a6531884e60977f59458e31dd99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } } }
I20260927 03:21:44.482124 5409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:44.482256 5410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c96a6531884e60977f59458e31dd99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c96a6531884e60977f59458e31dd99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35821 } } }
I20260927 03:21:44.482329 5410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:44.482436 5415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:44.482921 5415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:21:44.484443 5426 mvcc.cc:205] Tried to move back new op lower bound from 7333803231165890560 to 7333803231157776384. Current Snapshot: MvccSnapshot[applied={T|T < 7333803231165890560}]
I20260927 03:21:44.485195 5415 catalog_manager.cc:1347] Generated new cluster ID: 7f2fa08a97884fa29b9ec57610085ee4
I20260927 03:21:44.485249 5415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:44.488147 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:44.477091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54175 (local address 127.0.0.1:35821)
0927 03:21:44.477292 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:44.477300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:44.477446 (+ 146us) server_negotiation.cc:407] Connection header received
0927 03:21:44.478074 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:44.478082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:44.478143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:44.478294 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:44.478971 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:44.480396 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:44.481272 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:44.481899 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:44.482550 (+ 651us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:44.484939 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:44.484943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:44.484943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:44.484955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:44.486702 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:44.487071 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:44.487074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:44.487075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:44.487181 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:44.487381 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:44.487382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:44.487383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:44.487658 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:44.487800 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:44.487872 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:21:44.487961 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260927 03:21:44.489152 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:44.497265 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:44.490634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40137 (local address 127.0.0.1:35821)
0927 03:21:44.490670 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:44.490672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:44.490722 (+ 50us) server_negotiation.cc:407] Connection header received
0927 03:21:44.490784 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:44.490785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:44.490792 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:44.490840 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:44.491387 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:44.491842 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:44.492799 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:44.493088 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:44.493159 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:44.494444 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:44.494446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:44.494446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:44.494453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:44.495973 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:44.496383 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:44.496384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:44.496385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:44.496449 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:44.496696 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:44.496697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:44.496698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:44.496836 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:44.496956 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:44.497085 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 03:21:44.497147 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:21:44.504467 5415 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:44.505019 5415 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:44.507685 5415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c96a6531884e60977f59458e31dd99: Generated new TSK 0
I20260927 03:21:44.507866 5415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:44.514305 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5336
W20260927 03:21:44.519002 5321 connection.cc:535] client connection to 127.0.0.1:35821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35821): Connection reset by peer (error 104)
2026-09-27T03:21:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5435](info): set up 2 sockets
Sep 27 03:21:44 dist-test-slave-7j43 krb5kdc[5435](info): commencing operation
krb5kdc: starting...
W20260927 03:21:45.553161 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:45 dist-test-slave-7j43 krb5kdc[5435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:45.669173 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:45.748119 5448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:45.748358 5448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:45.748399 5448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:45.749576 5448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:45.749651 5448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:45.749676 5448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:45.749703 5448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:45.749733 5448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:45.751168 5448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:45.751443 5448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:45.751664 5448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:45.756430 5454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:45.756429 5453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:45.756486 5448 server_base.cc:1034] running on GCE node
W20260927 03:21:45.756435 5456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:45.756966 5448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:45.757269 5448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:45.758443 5448 hybrid_clock.cc:648] HybridClock initialized: now 1790479305758434 us; error 41 us; skew 500 ppm
Sep 27 03:21:45 dist-test-slave-7j43 krb5kdc[5435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:45.760835 5448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:45.761253 5448 webserver.cc:460] Webserver started at http://127.0.0.1:33647/ using document root <none> and password file <none>
I20260927 03:21:45.761475 5448 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:45.761545 5448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:45.761698 5448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:45.762710 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "c4b5108df20440c68ea7cf06def4dd9e"
format_stamp: "Formatted at 2026-09-27 03:21:45 on dist-test-slave-7j43"
server_key: "522792da8d209705ea5e18eb75fb2126"
server_key_iv: "d2deab17746501dec47fceb24913b3ff"
server_key_version: "encryptionkey@0"
I20260927 03:21:45.763063 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "c4b5108df20440c68ea7cf06def4dd9e"
format_stamp: "Formatted at 2026-09-27 03:21:45 on dist-test-slave-7j43"
server_key: "522792da8d209705ea5e18eb75fb2126"
server_key_iv: "d2deab17746501dec47fceb24913b3ff"
server_key_version: "encryptionkey@0"
I20260927 03:21:45.764410 5448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:45.765218 5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:45.765391 5448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:45.765489 5448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "c4b5108df20440c68ea7cf06def4dd9e"
format_stamp: "Formatted at 2026-09-27 03:21:45 on dist-test-slave-7j43"
server_key: "522792da8d209705ea5e18eb75fb2126"
server_key_iv: "d2deab17746501dec47fceb24913b3ff"
server_key_version: "encryptionkey@0"
I20260927 03:21:45.765576 5448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:45.784897 5448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:45.786706 5448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:45.786870 5448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:45.790923 5448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37083
I20260927 03:21:45.790966 5515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37083 every 8 connection(s)
I20260927 03:21:45.791399 5448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:45.792220 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:45.793825 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5448
I20260927 03:21:45.793964 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:45.794203 17618 external_mini_cluster.cc:1428] Setting key 780db8f0a70abd2fc07432c15fd10b0c
I20260927 03:21:45.795022 5516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e: Bootstrap starting.
I20260927 03:21:45.795646 5516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:45.795987 5516 log.cc:824] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:45.796741 5516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e: No bootstrap required, opened a new log
I20260927 03:21:45.799122 5516 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 03:21:45.799286 5516 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:45.799312 5516 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b5108df20440c68ea7cf06def4dd9e, State: Initialized, Role: FOLLOWER
I20260927 03:21:45.799443 5516 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 03:21:45.799512 5516 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:45.799533 5516 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:45.799599 5516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:45.800032 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:45.794767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60661 (local address 127.0.0.1:37083)
0927 03:21:45.794945 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:45.794952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:45.794976 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:21:45.795452 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:45.795460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:45.795507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:45.795623 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:45.795898 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:45.796931 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:45.798007 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:45.798390 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:45.798465 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:45.799183 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:45.799186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:45.799187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:45.799404 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:45.799460 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:45.799480 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:45.799578 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:45.799801 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 03:21:45.799830 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260927 03:21:45.800264 5516 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 03:21:45.800401 5516 leader_election.cc:302] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4b5108df20440c68ea7cf06def4dd9e; no voters:
I20260927 03:21:45.800547 5516 leader_election.cc:288] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:45.800616 5521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:45.800729 5521 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [term 1 LEADER]: Becoming Leader. State: Replica: c4b5108df20440c68ea7cf06def4dd9e, State: Running, Role: LEADER
I20260927 03:21:45.800801 5521 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 03:21:45.800802 5516 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:45.801082 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:45.801167 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4b5108df20440c68ea7cf06def4dd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260927 03:21:45.801296 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:45.801533 5528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:45.802078 5528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:45.802098 5522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4b5108df20440c68ea7cf06def4dd9e. Latest consensus state: current_term: 1 leader_uuid: "c4b5108df20440c68ea7cf06def4dd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b5108df20440c68ea7cf06def4dd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260927 03:21:45.802685 5522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:45.803777 5528 catalog_manager.cc:1347] Generated new cluster ID: 5931f46991f24c7683837b529222f8f8
I20260927 03:21:45.803838 5528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:45.806093 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:45.803033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34167 (local address 127.0.0.1:37083)
0927 03:21:45.803065 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:45.803067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:45.803163 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:21:45.803224 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:45.803225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:45.803235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:45.803283 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:45.803433 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:45.804046 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:45.804874 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:45.805090 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:45.805177 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:45.805835 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:45.805836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:45.805837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:45.805841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:45.805854 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:45.805858 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:45.805914 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:45.805992 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:21:45.806001 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:21:45.813285 5528 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:45.813979 5528 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:45.822099 5528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4b5108df20440c68ea7cf06def4dd9e: Generated new TSK 0
I20260927 03:21:45.822279 5528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:45.826473 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5448
W20260927 03:21:45.831122 5433 connection.cc:535] client connection to 127.0.0.1:37083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37083): Connection reset by peer (error 104)
2026-09-27T03:21:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:45 dist-test-slave-7j43 krb5kdc[5547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:45 dist-test-slave-7j43 krb5kdc[5547](info): set up 2 sockets
Sep 27 03:21:45 dist-test-slave-7j43 krb5kdc[5547](info): commencing operation
krb5kdc: starting...
W20260927 03:21:46.921415 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:46 dist-test-slave-7j43 krb5kdc[5547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:47.038744 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:47.118068 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:47.118256 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:47.118279 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:47.119339 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:47.119371 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:47.119385 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:47.119398 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:47.119410 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:47.120836 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:47.121013 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:47.121165 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:47.125715 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:47.125674 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:47.125833 5560 server_base.cc:1034] running on GCE node
W20260927 03:21:47.125856 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:47.126101 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:47.126374 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:47.127513 5560 hybrid_clock.cc:648] HybridClock initialized: now 1790479307127491 us; error 40 us; skew 500 ppm
Sep 27 03:21:47 dist-test-slave-7j43 krb5kdc[5547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:47.134205 5560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:47.134670 5560 webserver.cc:460] Webserver started at http://127.0.0.1:34527/ using document root <none> and password file <none>
I20260927 03:21:47.134917 5560 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:47.134990 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:47.135139 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:47.136142 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "43ac65fb96ec491482c9cba0dd2d20e9"
format_stamp: "Formatted at 2026-09-27 03:21:47 on dist-test-slave-7j43"
server_key: "fd03413dd6f41b2349b43a60496245da"
server_key_iv: "f9a2b8ca4d1a2e8e853fb7ab93a5e859"
server_key_version: "encryptionkey@0"
I20260927 03:21:47.136497 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "43ac65fb96ec491482c9cba0dd2d20e9"
format_stamp: "Formatted at 2026-09-27 03:21:47 on dist-test-slave-7j43"
server_key: "fd03413dd6f41b2349b43a60496245da"
server_key_iv: "f9a2b8ca4d1a2e8e853fb7ab93a5e859"
server_key_version: "encryptionkey@0"
I20260927 03:21:47.137847 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:47.138572 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:47.138783 5560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:21:47.138902 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "43ac65fb96ec491482c9cba0dd2d20e9"
format_stamp: "Formatted at 2026-09-27 03:21:47 on dist-test-slave-7j43"
server_key: "fd03413dd6f41b2349b43a60496245da"
server_key_iv: "f9a2b8ca4d1a2e8e853fb7ab93a5e859"
server_key_version: "encryptionkey@0"
I20260927 03:21:47.138981 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:47.142238 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:47.144421 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:47.144629 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:47.148698 5560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32885
I20260927 03:21:47.148753 5627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32885 every 8 connection(s)
I20260927 03:21:47.149075 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:47.149906 5628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:47.152009 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9: Bootstrap starting.
I20260927 03:21:47.152514 5628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:47.152856 5628 log.cc:824] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:47.153100 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5560
I20260927 03:21:47.153250 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:47.153491 17618 external_mini_cluster.cc:1428] Setting key d7296b17fcde3109639e104a63486ff0
I20260927 03:21:47.153714 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9: No bootstrap required, opened a new log
I20260927 03:21:47.155478 5628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260927 03:21:47.155635 5628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:47.155681 5628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ac65fb96ec491482c9cba0dd2d20e9, State: Initialized, Role: FOLLOWER
I20260927 03:21:47.155848 5628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260927 03:21:47.155943 5628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:47.155987 5628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:47.156060 5628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:47.156745 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:47.154159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51595 (local address 127.0.0.1:32885)
0927 03:21:47.154335 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:47.154342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:47.154366 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:21:47.155465 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:47.155471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:47.155496 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:47.155636 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:47.156258 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:47.156261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:47.156261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:47.156265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:47.156315 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:47.156327 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:47.156402 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:47.156471 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:21:47.156510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260927 03:21:47.157167 5628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260927 03:21:47.157275 5628 leader_election.cc:302] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43ac65fb96ec491482c9cba0dd2d20e9; no voters:
I20260927 03:21:47.157455 5628 leader_election.cc:288] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:47.157503 5633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:47.157601 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:47.157697 5628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:47.158910 5633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [term 1 LEADER]: Becoming Leader. State: Replica: 43ac65fb96ec491482c9cba0dd2d20e9, State: Running, Role: LEADER
I20260927 03:21:47.159011 5633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260927 03:21:47.159369 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260927 03:21:47.159468 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:47.159650 5639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ac65fb96ec491482c9cba0dd2d20e9. Latest consensus state: current_term: 1 leader_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ac65fb96ec491482c9cba0dd2d20e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260927 03:21:47.160025 5639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:47.160089 5646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:47.160643 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:47.158975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43391 (local address 127.0.0.1:32885)
0927 03:21:47.159008 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:47.159010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:47.159286 (+ 276us) server_negotiation.cc:407] Connection header received
0927 03:21:47.159355 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:47.159356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:47.159361 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:47.159725 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:47.160269 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:47.160270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:47.160271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:47.160274 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:47.160307 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:47.160313 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:47.160367 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:47.160552 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 03:21:47.160582 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:21:47.160653 5646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:47.162597 5646 catalog_manager.cc:1347] Generated new cluster ID: 51c812e446a84609a051093a82e1e78a
I20260927 03:21:47.162659 5646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:47.170459 5646 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:47.170622 5646 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:47.175824 5646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43ac65fb96ec491482c9cba0dd2d20e9: Generated new TSK 0
I20260927 03:21:47.176004 5646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:47.181629 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5560
2026-09-27T03:21:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:47 dist-test-slave-7j43 krb5kdc[5659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:47 dist-test-slave-7j43 krb5kdc[5659](info): set up 2 sockets
Sep 27 03:21:47 dist-test-slave-7j43 krb5kdc[5659](info): commencing operation
krb5kdc: starting...
W20260927 03:21:48.253136 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:48.371407 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:48.451102 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:48.451378 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:48.451421 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:48.452600 5672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:48.452673 5672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:48.452696 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:48.452723 5672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:48.452754 5672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:48.454238 5672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:48.454516 5672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:48.454749 5672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:48.459602 5677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:48.459609 5678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:48.459796 5672 server_base.cc:1034] running on GCE node
W20260927 03:21:48.459612 5680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:48.460131 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:48.460464 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:48.461632 5672 hybrid_clock.cc:648] HybridClock initialized: now 1790479308461625 us; error 38 us; skew 500 ppm
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:48.463989 5672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:48.464494 5672 webserver.cc:460] Webserver started at http://127.0.0.1:35691/ using document root <none> and password file <none>
I20260927 03:21:48.464789 5672 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:48.464876 5672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:48.465037 5672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:48.466230 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "f6de23376cea4727ab0e1f9e19ef9428"
format_stamp: "Formatted at 2026-09-27 03:21:48 on dist-test-slave-7j43"
server_key: "a6c7ede0d6cd5a8bd7c9a1de3738db6b"
server_key_iv: "de5942850c4734284a45efed70019af1"
server_key_version: "encryptionkey@0"
I20260927 03:21:48.466602 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "f6de23376cea4727ab0e1f9e19ef9428"
format_stamp: "Formatted at 2026-09-27 03:21:48 on dist-test-slave-7j43"
server_key: "a6c7ede0d6cd5a8bd7c9a1de3738db6b"
server_key_iv: "de5942850c4734284a45efed70019af1"
server_key_version: "encryptionkey@0"
I20260927 03:21:48.467918 5672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:48.468739 5687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:48.468994 5672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:48.469101 5672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "f6de23376cea4727ab0e1f9e19ef9428"
format_stamp: "Formatted at 2026-09-27 03:21:48 on dist-test-slave-7j43"
server_key: "a6c7ede0d6cd5a8bd7c9a1de3738db6b"
server_key_iv: "de5942850c4734284a45efed70019af1"
server_key_version: "encryptionkey@0"
I20260927 03:21:48.469173 5672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:48.487221 5672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:48.489657 5672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:48.489825 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:48.494071 5672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41729
I20260927 03:21:48.494132 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41729 every 8 connection(s)
I20260927 03:21:48.494505 5672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:48.495384 5740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:48.496034 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5672
I20260927 03:21:48.496146 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:48.496340 17618 external_mini_cluster.cc:1428] Setting key 8cedc7cafce770a1fde38bf41d12f141
I20260927 03:21:48.498172 5740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: Bootstrap starting.
I20260927 03:21:48.498747 5740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:48.499040 5740 log.cc:824] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:48.499729 5740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: No bootstrap required, opened a new log
I20260927 03:21:48.501837 5740 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260927 03:21:48.501986 5740 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:48.502031 5740 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6de23376cea4727ab0e1f9e19ef9428, State: Initialized, Role: FOLLOWER
I20260927 03:21:48.502156 5740 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260927 03:21:48.502241 5740 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:48.502285 5740 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:48.502332 5740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:48.502839 5740 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260927 03:21:48.502956 5740 leader_election.cc:302] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6de23376cea4727ab0e1f9e19ef9428; no voters:
I20260927 03:21:48.503134 5740 leader_election.cc:288] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:48.503177 5745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:21:48.503290 5745 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [term 1 LEADER]: Becoming Leader. State: Replica: f6de23376cea4727ab0e1f9e19ef9428, State: Running, Role: LEADER
I20260927 03:21:48.503369 5745 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260927 03:21:48.503396 5740 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:48.505056 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6de23376cea4727ab0e1f9e19ef9428. Latest consensus state: current_term: 1 leader_uuid: "f6de23376cea4727ab0e1f9e19ef9428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } } }
I20260927 03:21:48.505210 5746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6de23376cea4727ab0e1f9e19ef9428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6de23376cea4727ab0e1f9e19ef9428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } } }
I20260927 03:21:48.505273 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:48.505287 5746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428 [sys.catalog]: This master's current role is: LEADER
W20260927 03:21:48.505422 5759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:21:48.505503 5759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:21:48.505576 5763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:48.505731 5763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:48.507576 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:48.496942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44997 (local address 127.0.0.1:41729)
0927 03:21:48.497125 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:48.497134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:48.497172 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:21:48.497564 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:48.497571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:48.497617 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:48.497771 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:48.498443 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:48.499719 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:48.500601 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:48.501021 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:48.501144 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:48.503734 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:48.503738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:48.503738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:48.503749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:48.506090 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:48.506491 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:48.506493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:48.506493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:48.506567 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:48.506786 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:48.506786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:48.506787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:48.506991 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:48.507092 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:48.507245 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:21:48.507375 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260927 03:21:48.508598 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:48.509555 5763 catalog_manager.cc:1347] Generated new cluster ID: 87b7687a5d524776ba778b65de71a056
I20260927 03:21:48.509618 5763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:48.516541 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:48.510070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46805 (local address 127.0.0.1:41729)
0927 03:21:48.510105 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:48.510107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:48.510204 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:21:48.510269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:48.510270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:48.510277 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:48.510316 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:48.510866 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:48.511335 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:48.512299 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:48.512490 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:48.512537 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:48.513848 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:48.513850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:48.513851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:48.513859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:48.515425 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:48.515786 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:48.515787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:48.515787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:48.515855 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:48.516048 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:48.516049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:48.516049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:48.516170 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:48.516275 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:48.516374 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:21:48.516446 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:21:48.525714 5763 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:48.526227 5763 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:48.531721 5763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6de23376cea4727ab0e1f9e19ef9428: Generated new TSK 0
I20260927 03:21:48.531925 5763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:48.534379 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5672
2026-09-27T03:21:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5772](info): set up 2 sockets
Sep 27 03:21:48 dist-test-slave-7j43 krb5kdc[5772](info): commencing operation
krb5kdc: starting...
W20260927 03:21:49.565121 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:49 dist-test-slave-7j43 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:49.685441 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:49.764811 5785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:49.765002 5785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:49.765022 5785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:49.766067 5785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:49.766103 5785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:49.766116 5785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:49.766129 5785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:49.766140 5785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:49.767489 5785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:49.767666 5785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:49.767818 5785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:49.772514 5791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:49.772548 5793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:49.772606 5785 server_base.cc:1034] running on GCE node
W20260927 03:21:49.772552 5790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:49.773020 5785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:49.773315 5785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:49.774459 5785 hybrid_clock.cc:648] HybridClock initialized: now 1790479309774445 us; error 28 us; skew 500 ppm
Sep 27 03:21:49 dist-test-slave-7j43 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:49.777078 5785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:49.777479 5785 webserver.cc:460] Webserver started at http://127.0.0.1:41259/ using document root <none> and password file <none>
I20260927 03:21:49.777714 5785 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:49.777760 5785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:49.777926 5785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:49.778921 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "60d7a10c95db4494861c4d21dc4405c2"
format_stamp: "Formatted at 2026-09-27 03:21:49 on dist-test-slave-7j43"
server_key: "ef8166fd5b8628d3c4da8aa6f633cf95"
server_key_iv: "8e6632ef650c8a8aa052fad9d546c977"
server_key_version: "encryptionkey@0"
I20260927 03:21:49.779290 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "60d7a10c95db4494861c4d21dc4405c2"
format_stamp: "Formatted at 2026-09-27 03:21:49 on dist-test-slave-7j43"
server_key: "ef8166fd5b8628d3c4da8aa6f633cf95"
server_key_iv: "8e6632ef650c8a8aa052fad9d546c977"
server_key_version: "encryptionkey@0"
I20260927 03:21:49.780561 5785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:21:49.781358 5800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:49.781545 5785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:49.781644 5785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "60d7a10c95db4494861c4d21dc4405c2"
format_stamp: "Formatted at 2026-09-27 03:21:49 on dist-test-slave-7j43"
server_key: "ef8166fd5b8628d3c4da8aa6f633cf95"
server_key_iv: "8e6632ef650c8a8aa052fad9d546c977"
server_key_version: "encryptionkey@0"
I20260927 03:21:49.781737 5785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:49.793095 5785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:49.795357 5785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:49.795464 5785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:49.799516 5785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42693
I20260927 03:21:49.799639 5852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42693 every 8 connection(s)
I20260927 03:21:49.799928 5785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:49.800804 5853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:49.802974 5853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2: Bootstrap starting.
I20260927 03:21:49.803535 5853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:49.803821 5853 log.cc:824] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:49.804541 5853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2: No bootstrap required, opened a new log
I20260927 03:21:49.806264 5853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260927 03:21:49.806394 5853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:49.806418 5853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7a10c95db4494861c4d21dc4405c2, State: Initialized, Role: FOLLOWER
I20260927 03:21:49.806597 5853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260927 03:21:49.806660 5853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:49.806702 5853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:49.806754 5853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:49.807257 5853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260927 03:21:49.807341 5853 leader_election.cc:302] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d7a10c95db4494861c4d21dc4405c2; no voters:
I20260927 03:21:49.807518 5853 leader_election.cc:288] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:49.807564 5856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:49.807750 5856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [term 1 LEADER]: Becoming Leader. State: Replica: 60d7a10c95db4494861c4d21dc4405c2, State: Running, Role: LEADER
I20260927 03:21:49.807782 5853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:49.807839 5856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260927 03:21:49.808235 5858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d7a10c95db4494861c4d21dc4405c2. Latest consensus state: current_term: 1 leader_uuid: "60d7a10c95db4494861c4d21dc4405c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } } }
I20260927 03:21:49.808324 5858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:49.808358 5857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d7a10c95db4494861c4d21dc4405c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7a10c95db4494861c4d21dc4405c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } } }
I20260927 03:21:49.808427 5857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:49.808560 5865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:49.809139 5865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:49.809859 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5785
I20260927 03:21:49.809979 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:49.810163 17618 external_mini_cluster.cc:1428] Setting key c5ab4cd771ac02f9eef0a08cdc19e5bf
I20260927 03:21:49.810890 5865 catalog_manager.cc:1347] Generated new cluster ID: ab2136efc2104c43a75bdeabf50b34fb
I20260927 03:21:49.810940 5865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:49.817857 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:49.810610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40525 (local address 127.0.0.1:42693)
0927 03:21:49.810763 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:49.810772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:49.810812 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:21:49.812767 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:49.812774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:49.812836 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:49.812958 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:49.813215 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:49.814611 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:49.815792 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:49.816379 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:49.816470 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:49.817200 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:49.817204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:49.817204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:49.817393 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:49.817427 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:49.817441 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:49.817500 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:49.817600 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:21:49.817655 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260927 03:21:49.818868 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:49.823874 5865 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:49.824445 5865 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:49.827284 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:49.820324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51853 (local address 127.0.0.1:42693)
0927 03:21:49.820357 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:49.820359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:49.823783 (+ 3424us) server_negotiation.cc:407] Connection header received
0927 03:21:49.823893 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:49.823893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:49.823906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:49.823966 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:49.824158 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:49.824951 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:49.825937 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:49.826277 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:49.826348 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:49.827015 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:49.827017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:49.827017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:49.827022 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:49.827037 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:49.827041 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:49.827097 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:49.827165 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:21:49.827204 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:21:49.827497 5865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d7a10c95db4494861c4d21dc4405c2: Generated new TSK 0
I20260927 03:21:49.827656 5865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:49.834890 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5785
2026-09-27T03:21:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:49 dist-test-slave-7j43 krb5kdc[5884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:49 dist-test-slave-7j43 krb5kdc[5884](info): set up 2 sockets
Sep 27 03:21:49 dist-test-slave-7j43 krb5kdc[5884](info): commencing operation
krb5kdc: starting...
W20260927 03:21:50.869238 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:50 dist-test-slave-7j43 krb5kdc[5884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:50.983029 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:51.061892 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:51.062121 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:51.062161 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:51.063288 5897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:51.063354 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:51.063378 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:51.063405 5897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:51.063436 5897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:51.064931 5897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:51.065188 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:51.065412 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:51.070015 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:51.070115 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:51.070132 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:51.070482 5897 server_base.cc:1034] running on GCE node
I20260927 03:21:51.070684 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:51.070922 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:51.072070 5897 hybrid_clock.cc:648] HybridClock initialized: now 1790479311072056 us; error 28 us; skew 500 ppm
Sep 27 03:21:51 dist-test-slave-7j43 krb5kdc[5884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:51.074602 5897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:51.075043 5897 webserver.cc:460] Webserver started at http://127.0.0.1:40139/ using document root <none> and password file <none>
I20260927 03:21:51.075239 5897 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:51.075280 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:51.075366 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:51.076320 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "7a28bc151c0d46ddb7a19df67e411ccb"
format_stamp: "Formatted at 2026-09-27 03:21:51 on dist-test-slave-7j43"
server_key: "fee4d53bd801753b3f12af1cb67d4586"
server_key_iv: "17d53c34e8e9dc330fbe937abf80d31d"
server_key_version: "encryptionkey@0"
I20260927 03:21:51.076666 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "7a28bc151c0d46ddb7a19df67e411ccb"
format_stamp: "Formatted at 2026-09-27 03:21:51 on dist-test-slave-7j43"
server_key: "fee4d53bd801753b3f12af1cb67d4586"
server_key_iv: "17d53c34e8e9dc330fbe937abf80d31d"
server_key_version: "encryptionkey@0"
I20260927 03:21:51.077935 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:51.078634 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:51.078805 5897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:51.078908 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "7a28bc151c0d46ddb7a19df67e411ccb"
format_stamp: "Formatted at 2026-09-27 03:21:51 on dist-test-slave-7j43"
server_key: "fee4d53bd801753b3f12af1cb67d4586"
server_key_iv: "17d53c34e8e9dc330fbe937abf80d31d"
server_key_version: "encryptionkey@0"
I20260927 03:21:51.078980 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:51.082258 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:51.084458 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:51.084652 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:51.088748 5897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38959
I20260927 03:21:51.088863 5964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38959 every 8 connection(s)
I20260927 03:21:51.089143 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:51.089975 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:51.092134 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb: Bootstrap starting.
I20260927 03:21:51.092770 5965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:51.093057 5965 log.cc:824] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:51.093711 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb: No bootstrap required, opened a new log
I20260927 03:21:51.095283 5965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 03:21:51.095418 5965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:51.095441 5965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a28bc151c0d46ddb7a19df67e411ccb, State: Initialized, Role: FOLLOWER
I20260927 03:21:51.095592 5965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 03:21:51.095652 5965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:51.095674 5965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:51.095738 5965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:51.096251 5965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 03:21:51.096362 5965 leader_election.cc:302] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a28bc151c0d46ddb7a19df67e411ccb; no voters:
I20260927 03:21:51.096539 5965 leader_election.cc:288] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:51.096618 5968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:51.096771 5968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [term 1 LEADER]: Becoming Leader. State: Replica: 7a28bc151c0d46ddb7a19df67e411ccb, State: Running, Role: LEADER
I20260927 03:21:51.096854 5965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:51.096952 5968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 03:21:51.097278 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 5897
I20260927 03:21:51.097312 5969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } } }
I20260927 03:21:51.097414 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:51.097419 5969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:51.097323 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a28bc151c0d46ddb7a19df67e411ccb. Latest consensus state: current_term: 1 leader_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a28bc151c0d46ddb7a19df67e411ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } } }
I20260927 03:21:51.097599 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:51.097618 17618 external_mini_cluster.cc:1428] Setting key d4ceff11f22b5f11153885369c576fac
I20260927 03:21:51.097626 5972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:51.098115 5972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:51.100075 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:51.098242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34213 (local address 127.0.0.1:38959)
0927 03:21:51.098508 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:51.098517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:51.098548 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:21:51.098719 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:51.098727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:51.098759 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:51.098898 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:51.099526 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:51.099529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:51.099530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:51.099534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:51.099587 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:51.099598 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:51.099679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:51.099884 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 03:21:51.099928 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260927 03:21:51.100976 5972 catalog_manager.cc:1347] Generated new cluster ID: cf792d1bbbb54f65b2aca4b73843ed84
I20260927 03:21:51.101027 5972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:51.101086 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:51.106012 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:51.104856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43505 (local address 127.0.0.1:38959)
0927 03:21:51.104892 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:51.104894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:51.105050 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:21:51.105131 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:51.105132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:51.105138 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:51.105192 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:51.105801 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:51.105802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:51.105802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:51.105806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:51.105821 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:51.105825 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:51.105871 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:51.105933 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 03:21:51.105949 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:21:51.111876 5972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:51.112044 5972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:51.120208 5972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a28bc151c0d46ddb7a19df67e411ccb: Generated new TSK 0
I20260927 03:21:51.120406 5972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:51.123597 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 5897
2026-09-27T03:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:51 dist-test-slave-7j43 krb5kdc[5996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:51 dist-test-slave-7j43 krb5kdc[5996](info): set up 2 sockets
Sep 27 03:21:51 dist-test-slave-7j43 krb5kdc[5996](info): commencing operation
krb5kdc: starting...
W20260927 03:21:52.149168 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[5996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:52.263301 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:52.343865 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:52.344095 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:52.344132 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:52.345291 6009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:52.345360 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:52.345384 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:52.345418 6009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:52.345445 6009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:52.346886 6009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:52.347146 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:52.347364 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:52.352128 6017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:52.352114 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:52.352155 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:52.352257 6009 server_base.cc:1034] running on GCE node
I20260927 03:21:52.352622 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:52.352945 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:52.354111 6009 hybrid_clock.cc:648] HybridClock initialized: now 1790479312354088 us; error 35 us; skew 500 ppm
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[5996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:52.361001 6009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:52.361442 6009 webserver.cc:460] Webserver started at http://127.0.0.1:36847/ using document root <none> and password file <none>
I20260927 03:21:52.361635 6009 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:52.361677 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:52.361771 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:52.362792 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "2ede29cbed4f41648324e118ac377824"
format_stamp: "Formatted at 2026-09-27 03:21:52 on dist-test-slave-7j43"
server_key: "17f447d44b6298244e92f84dcd5372d4"
server_key_iv: "94efa5754d036ab56707607a148e0a7c"
server_key_version: "encryptionkey@0"
I20260927 03:21:52.363101 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "2ede29cbed4f41648324e118ac377824"
format_stamp: "Formatted at 2026-09-27 03:21:52 on dist-test-slave-7j43"
server_key: "17f447d44b6298244e92f84dcd5372d4"
server_key_iv: "94efa5754d036ab56707607a148e0a7c"
server_key_version: "encryptionkey@0"
I20260927 03:21:52.364384 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:21:52.365242 6024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:52.365486 6009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:21:52.365562 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "2ede29cbed4f41648324e118ac377824"
format_stamp: "Formatted at 2026-09-27 03:21:52 on dist-test-slave-7j43"
server_key: "17f447d44b6298244e92f84dcd5372d4"
server_key_iv: "94efa5754d036ab56707607a148e0a7c"
server_key_version: "encryptionkey@0"
I20260927 03:21:52.365654 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:52.385537 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:52.387501 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:52.387676 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:52.391546 6009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46471
I20260927 03:21:52.391582 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46471 every 8 connection(s)
I20260927 03:21:52.391950 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:52.392916 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:52.395133 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824: Bootstrap starting.
I20260927 03:21:52.395711 6077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:52.395994 6077 log.cc:824] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:52.396704 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824: No bootstrap required, opened a new log
I20260927 03:21:52.397883 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6009
I20260927 03:21:52.397981 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:52.398217 17618 external_mini_cluster.cc:1428] Setting key 3dde6dfe6148b20e64b8d267e77958fe
I20260927 03:21:52.398480 6077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 03:21:52.398636 6077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:52.398671 6077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ede29cbed4f41648324e118ac377824, State: Initialized, Role: FOLLOWER
I20260927 03:21:52.398797 6077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 03:21:52.398922 6077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:52.398970 6077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:52.399012 6077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:52.399752 6077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 03:21:52.399873 6077 leader_election.cc:302] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ede29cbed4f41648324e118ac377824; no voters:
I20260927 03:21:52.400072 6077 leader_election.cc:288] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:52.400101 6082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:52.400213 6082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [term 1 LEADER]: Becoming Leader. State: Replica: 2ede29cbed4f41648324e118ac377824, State: Running, Role: LEADER
I20260927 03:21:52.400347 6077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:52.400370 6082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 03:21:52.400797 6082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ede29cbed4f41648324e118ac377824. Latest consensus state: current_term: 1 leader_uuid: "2ede29cbed4f41648324e118ac377824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } } }
I20260927 03:21:52.400804 6083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ede29cbed4f41648324e118ac377824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ede29cbed4f41648324e118ac377824" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } } }
I20260927 03:21:52.400972 6082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:52.400980 6083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:52.401263 6091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:52.401821 6091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:52.404034 6091 catalog_manager.cc:1347] Generated new cluster ID: 1f1d4f7dd2bd4b429aa24ceddb06bcca
I20260927 03:21:52.404096 6091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[5996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:21:52.410002 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:52.398732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:51111 (local address 127.0.0.1:46471)
0927 03:21:52.398912 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:52.398922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:52.398958 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:21:52.399085 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:52.399092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:52.399153 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:52.399301 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:52.399903 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:52.401293 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:52.402371 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:52.402846 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:52.402978 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:52.405701 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:52.405704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:52.405705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:52.405716 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:52.407671 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:52.408084 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:52.408087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:52.408087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:52.408202 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:52.408433 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:52.408434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:52.408434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:52.408760 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:52.409416 (+ 656us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:52.409506 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 03:21:52.409623 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260927 03:21:52.411248 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:52.415185 6091 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:52.415688 6091 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:52.420248 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:52.412750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:45387 (local address 127.0.0.1:46471)
0927 03:21:52.412783 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:52.412785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:52.412896 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:21:52.412989 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:52.412990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:52.412999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:52.413028 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:52.413734 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:52.414549 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:52.415579 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:52.415903 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:52.415942 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:52.417402 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:52.417403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:52.417404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:52.417410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:52.418953 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:52.419380 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:52.419381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:52.419381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:52.419432 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:52.419663 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:52.419664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:52.419664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:52.419779 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:52.419848 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:52.420073 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 03:21:52.420135 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":16}
I20260927 03:21:52.421648 6091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ede29cbed4f41648324e118ac377824: Generated new TSK 0
I20260927 03:21:52.421813 6091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:52.426826 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6009
W20260927 03:21:52.431263 5994 connection.cc:535] client connection to 127.0.0.1:46471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46471): Connection reset by peer (error 104)
2026-09-27T03:21:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[6108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[6108](info): set up 2 sockets
Sep 27 03:21:52 dist-test-slave-7j43 krb5kdc[6108](info): commencing operation
krb5kdc: starting...
W20260927 03:21:53.449631 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:53 dist-test-slave-7j43 krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:53.569602 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:53.653009 6121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:53.653263 6121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:53.653306 6121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:53.654449 6121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:53.654520 6121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:53.654548 6121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:53.654580 6121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:53.654615 6121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:53.656066 6121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:53.656334 6121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:53.656589 6121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:53.661325 6126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:53.661334 6127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:53.661370 6121 server_base.cc:1034] running on GCE node
W20260927 03:21:53.661334 6129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:53.661716 6121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:53.662060 6121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:53.663215 6121 hybrid_clock.cc:648] HybridClock initialized: now 1790479313663203 us; error 41 us; skew 500 ppm
Sep 27 03:21:53 dist-test-slave-7j43 krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:53.665769 6121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:53.666222 6121 webserver.cc:460] Webserver started at http://127.0.0.1:38579/ using document root <none> and password file <none>
I20260927 03:21:53.666450 6121 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:53.666520 6121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:53.666684 6121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:53.667727 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "598943187d5940c7b6ce2b87465bda43"
format_stamp: "Formatted at 2026-09-27 03:21:53 on dist-test-slave-7j43"
server_key: "6d9197fdeb1ac5001abf62ada9f02026"
server_key_iv: "29556df013bf0aa5cd155c5218f941ad"
server_key_version: "encryptionkey@0"
I20260927 03:21:53.668105 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "598943187d5940c7b6ce2b87465bda43"
format_stamp: "Formatted at 2026-09-27 03:21:53 on dist-test-slave-7j43"
server_key: "6d9197fdeb1ac5001abf62ada9f02026"
server_key_iv: "29556df013bf0aa5cd155c5218f941ad"
server_key_version: "encryptionkey@0"
I20260927 03:21:53.669586 6121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:21:53.670341 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:53.670512 6121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:53.670624 6121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "598943187d5940c7b6ce2b87465bda43"
format_stamp: "Formatted at 2026-09-27 03:21:53 on dist-test-slave-7j43"
server_key: "6d9197fdeb1ac5001abf62ada9f02026"
server_key_iv: "29556df013bf0aa5cd155c5218f941ad"
server_key_version: "encryptionkey@0"
I20260927 03:21:53.670704 6121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:53.680984 6121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:53.682852 6121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:53.683019 6121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:53.686951 6121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43317
I20260927 03:21:53.687002 6188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43317 every 8 connection(s)
I20260927 03:21:53.687354 6121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:53.688259 6189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:53.690650 6189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: Bootstrap starting.
I20260927 03:21:53.691248 6189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:53.691545 6189 log.cc:824] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:53.692233 6189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: No bootstrap required, opened a new log
I20260927 03:21:53.693975 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6121
I20260927 03:21:53.694065 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:53.693956 6189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260927 03:21:53.694165 6189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:53.694232 6189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598943187d5940c7b6ce2b87465bda43, State: Initialized, Role: FOLLOWER
I20260927 03:21:53.694280 17618 external_mini_cluster.cc:1428] Setting key 47bbbdd7c130ef2a3095488783da0a0c
I20260927 03:21:53.694398 6189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260927 03:21:53.694499 6189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:53.694535 6189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:53.694590 6189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:53.695748 6189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260927 03:21:53.695858 6189 leader_election.cc:302] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 598943187d5940c7b6ce2b87465bda43; no voters:
I20260927 03:21:53.696024 6189 leader_election.cc:288] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:53.696054 6194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:53.696218 6194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [term 1 LEADER]: Becoming Leader. State: Replica: 598943187d5940c7b6ce2b87465bda43, State: Running, Role: LEADER
I20260927 03:21:53.696292 6189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:53.696358 6194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260927 03:21:53.696729 6194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 598943187d5940c7b6ce2b87465bda43. Latest consensus state: current_term: 1 leader_uuid: "598943187d5940c7b6ce2b87465bda43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260927 03:21:53.696947 6194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:53.696866 6196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "598943187d5940c7b6ce2b87465bda43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598943187d5940c7b6ce2b87465bda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260927 03:21:53.698021 6196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43 [sys.catalog]: This master's current role is: LEADER
W20260927 03:21:53.698104 6209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:21:53.698181 6209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:21:53.698266 6210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:53.698472 6210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:53.700022 6210 catalog_manager.cc:1347] Generated new cluster ID: b5c863c42dc642c5ba8bdc0e06175dc6
I20260927 03:21:53.700075 6210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:53.700516 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:53.694753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:37983 (local address 127.0.0.1:43317)
0927 03:21:53.695005 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:53.695014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:53.695051 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:21:53.695185 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:53.695194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:53.695266 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:53.695449 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:53.695598 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:53.697024 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:53.698156 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:53.698730 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:53.698867 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:53.699762 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:53.699766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:53.699766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:53.699965 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:53.700007 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:53.700029 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:53.700123 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:53.700271 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:21:53.700304 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":125,"threads_started":1}
I20260927 03:21:53.705338 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:53.709772 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:53.706719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:58471 (local address 127.0.0.1:43317)
0927 03:21:53.706755 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:53.706758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:53.706883 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:21:53.706960 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:53.706961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:53.706984 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:53.707059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:53.707206 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:53.707695 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:53.708435 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:53.708724 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:53.708810 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:53.709523 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:53.709524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:53.709525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:53.709530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:53.709544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:53.709548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:53.709602 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:53.709662 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:21:53.709674 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:21:53.712447 6210 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:53.713176 6210 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:53.718532 6210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 598943187d5940c7b6ce2b87465bda43: Generated new TSK 0
I20260927 03:21:53.718703 6210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:53.722640 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6121
W20260927 03:21:53.726955 6106 connection.cc:535] client connection to 127.0.0.1:43317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43317): Connection reset by peer (error 104)
2026-09-27T03:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:53 dist-test-slave-7j43 krb5kdc[6220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:53 dist-test-slave-7j43 krb5kdc[6220](info): set up 2 sockets
Sep 27 03:21:53 dist-test-slave-7j43 krb5kdc[6220](info): commencing operation
krb5kdc: starting...
W20260927 03:21:54.753134 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:54 dist-test-slave-7j43 krb5kdc[6220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:54.867381 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:54.946769 6233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:54.947022 6233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:54.947062 6233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:54.948202 6233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:54.948271 6233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:54.948294 6233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:54.948324 6233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:54.948350 6233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:54.949875 6233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:54.950150 6233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:54.950373 6233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:21:54.955220 6233 server_base.cc:1034] running on GCE node
W20260927 03:21:54.955255 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:54.955224 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:54.955363 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:54.955636 6233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:54.955929 6233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:54.957098 6233 hybrid_clock.cc:648] HybridClock initialized: now 1790479314957084 us; error 42 us; skew 500 ppm
Sep 27 03:21:54 dist-test-slave-7j43 krb5kdc[6220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:54.959412 6233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:54.959844 6233 webserver.cc:460] Webserver started at http://127.0.0.1:46091/ using document root <none> and password file <none>
I20260927 03:21:54.960033 6233 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:54.960073 6233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:54.960157 6233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:54.961185 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "4e6aa77f7c99476982554fd12cc0d79f"
format_stamp: "Formatted at 2026-09-27 03:21:54 on dist-test-slave-7j43"
server_key: "fb53dc02b0e15987e5aa10b46b6cc11e"
server_key_iv: "cc652ae5df8ac28d4c4d8f1bc3a10047"
server_key_version: "encryptionkey@0"
I20260927 03:21:54.961483 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "4e6aa77f7c99476982554fd12cc0d79f"
format_stamp: "Formatted at 2026-09-27 03:21:54 on dist-test-slave-7j43"
server_key: "fb53dc02b0e15987e5aa10b46b6cc11e"
server_key_iv: "cc652ae5df8ac28d4c4d8f1bc3a10047"
server_key_version: "encryptionkey@0"
I20260927 03:21:54.962679 6233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:21:54.963343 6248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:54.963529 6233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:21:54.963608 6233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "4e6aa77f7c99476982554fd12cc0d79f"
format_stamp: "Formatted at 2026-09-27 03:21:54 on dist-test-slave-7j43"
server_key: "fb53dc02b0e15987e5aa10b46b6cc11e"
server_key_iv: "cc652ae5df8ac28d4c4d8f1bc3a10047"
server_key_version: "encryptionkey@0"
I20260927 03:21:54.963660 6233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:54.966804 6233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:54.968954 6233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:54.969058 6233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:54.972740 6233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34311
I20260927 03:21:54.972815 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34311 every 8 connection(s)
I20260927 03:21:54.973120 6233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:54.973949 6301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:54.976028 6301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f: Bootstrap starting.
I20260927 03:21:54.976637 6301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:54.976913 6301 log.cc:824] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:54.977564 6301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f: No bootstrap required, opened a new log
I20260927 03:21:54.979103 6301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } }
I20260927 03:21:54.979255 6301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:54.979303 6301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6aa77f7c99476982554fd12cc0d79f, State: Initialized, Role: FOLLOWER
I20260927 03:21:54.979480 6301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } }
I20260927 03:21:54.979578 6301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:54.979619 6301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:54.979669 6301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:54.980175 6301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } }
I20260927 03:21:54.980288 6301 leader_election.cc:302] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e6aa77f7c99476982554fd12cc0d79f; no voters:
I20260927 03:21:54.980463 6301 leader_election.cc:288] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:54.980491 6304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:54.980705 6304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [term 1 LEADER]: Becoming Leader. State: Replica: 4e6aa77f7c99476982554fd12cc0d79f, State: Running, Role: LEADER
I20260927 03:21:54.980733 6301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:54.980827 6304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } }
I20260927 03:21:54.981174 6305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e6aa77f7c99476982554fd12cc0d79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } } }
I20260927 03:21:54.981184 6306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e6aa77f7c99476982554fd12cc0d79f. Latest consensus state: current_term: 1 leader_uuid: "4e6aa77f7c99476982554fd12cc0d79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6aa77f7c99476982554fd12cc0d79f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34311 } } }
I20260927 03:21:54.981272 6305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:54.981328 6306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:54.981534 6308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:54.981895 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6233
I20260927 03:21:54.982023 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:54.982038 6308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:54.982210 17618 external_mini_cluster.cc:1428] Setting key d179f6289acb73adcf803a9e4146eb34
I20260927 03:21:54.984028 6308 catalog_manager.cc:1347] Generated new cluster ID: b526baa5e2054012b70e2a557e29fda9
I20260927 03:21:54.984084 6308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:54.984809 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:54.982803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:51547 (local address 127.0.0.1:34311)
0927 03:21:54.983260 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:54.983268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:54.983313 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:21:54.983444 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:54.983451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:54.983480 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:54.983621 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:54.984339 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:54.984343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:54.984343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:54.984347 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:54.984390 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:54.984401 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:54.984468 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:54.984548 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:21:54.984637 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":81,"threads_started":1}
I20260927 03:21:54.985656 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:54.990080 6308 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:54.990226 6308 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:54.991155 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:54.990012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:44021 (local address 127.0.0.1:34311)
0927 03:21:54.990041 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:54.990043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:54.990208 (+ 165us) server_negotiation.cc:407] Connection header received
0927 03:21:54.990257 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:54.990258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:54.990263 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:54.990323 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:54.990942 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:54.990944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:54.990944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:54.990948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:54.990962 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:54.990966 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:54.991020 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:54.991087 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:21:54.991099 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 03:21:54.996436 6308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e6aa77f7c99476982554fd12cc0d79f: Generated new TSK 0
I20260927 03:21:54.996611 6308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:54.999842 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6233
2026-09-27T03:21:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:55 dist-test-slave-7j43 krb5kdc[6332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:55 dist-test-slave-7j43 krb5kdc[6332](info): set up 2 sockets
Sep 27 03:21:55 dist-test-slave-7j43 krb5kdc[6332](info): commencing operation
krb5kdc: starting...
W20260927 03:21:56.025173 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:56.139874 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:56.219937 6345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:56.220161 6345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:56.220197 6345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:56.221380 6345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:56.221444 6345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:56.221472 6345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:56.221514 6345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:56.221544 6345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:56.222998 6345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:56.223272 6345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:56.223484 6345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:56.228267 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:56.228236 6351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:56.228250 6353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:56.228552 6345 server_base.cc:1034] running on GCE node
I20260927 03:21:56.228852 6345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:56.229156 6345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:56.230330 6345 hybrid_clock.cc:648] HybridClock initialized: now 1790479316230325 us; error 36 us; skew 500 ppm
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:56.232838 6345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:56.233258 6345 webserver.cc:460] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
I20260927 03:21:56.233487 6345 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:56.233561 6345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:56.233711 6345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:56.234779 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "5f054fa9aa194eb5936a458ccb241267"
format_stamp: "Formatted at 2026-09-27 03:21:56 on dist-test-slave-7j43"
server_key: "68dc609659301435277e187082299de6"
server_key_iv: "7c230330811061b9a25b4eefcc7dc3a4"
server_key_version: "encryptionkey@0"
I20260927 03:21:56.235138 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "5f054fa9aa194eb5936a458ccb241267"
format_stamp: "Formatted at 2026-09-27 03:21:56 on dist-test-slave-7j43"
server_key: "68dc609659301435277e187082299de6"
server_key_iv: "7c230330811061b9a25b4eefcc7dc3a4"
server_key_version: "encryptionkey@0"
I20260927 03:21:56.236469 6345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:21:56.237287 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:56.237464 6345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:21:56.237564 6345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "5f054fa9aa194eb5936a458ccb241267"
format_stamp: "Formatted at 2026-09-27 03:21:56 on dist-test-slave-7j43"
server_key: "68dc609659301435277e187082299de6"
server_key_iv: "7c230330811061b9a25b4eefcc7dc3a4"
server_key_version: "encryptionkey@0"
I20260927 03:21:56.237646 6345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:56.246729 6345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:56.248478 6345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:56.248694 6345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:56.252661 6345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46349
I20260927 03:21:56.252710 6412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46349 every 8 connection(s)
I20260927 03:21:56.253069 6345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:56.254010 6413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:56.254547 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6345
I20260927 03:21:56.254676 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:56.254870 17618 external_mini_cluster.cc:1428] Setting key 42f64abc731a3e1f0d54325aa803b7cc
I20260927 03:21:56.257890 6413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267: Bootstrap starting.
I20260927 03:21:56.258553 6413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:56.258844 6413 log.cc:824] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:56.259545 6413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267: No bootstrap required, opened a new log
I20260927 03:21:56.261543 6413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 03:21:56.261720 6413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:56.261756 6413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f054fa9aa194eb5936a458ccb241267, State: Initialized, Role: FOLLOWER
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:21:56.261883 6413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 03:21:56.261976 6413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:56.262037 6413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:56.262090 6413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:56.262620 6413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 03:21:56.262727 6413 leader_election.cc:302] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f054fa9aa194eb5936a458ccb241267; no voters:
I20260927 03:21:56.262905 6413 leader_election.cc:288] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:56.262936 6418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:56.263154 6418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [term 1 LEADER]: Becoming Leader. State: Replica: 5f054fa9aa194eb5936a458ccb241267, State: Running, Role: LEADER
I20260927 03:21:56.263159 6413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:56.263381 6418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 03:21:56.263741 6420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f054fa9aa194eb5936a458ccb241267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260927 03:21:56.263813 6420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:56.264031 6425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:56.264508 6425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:56.265219 6419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f054fa9aa194eb5936a458ccb241267. Latest consensus state: current_term: 1 leader_uuid: "5f054fa9aa194eb5936a458ccb241267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f054fa9aa194eb5936a458ccb241267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260927 03:21:56.265300 6419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:56.266141 6425 catalog_manager.cc:1347] Generated new cluster ID: 1648a29621f645f4b0cb5c89b6817c8a
I20260927 03:21:56.266193 6425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:56.266396 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:56.255374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:50691 (local address 127.0.0.1:46349)
0927 03:21:56.255544 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:56.255554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:56.255586 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:21:56.256097 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:56.256106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:56.256173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:56.256327 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:56.256986 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:56.258344 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:56.259138 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:56.259596 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:56.259717 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:56.262400 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:56.262403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:56.262403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:56.262414 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:56.264484 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:56.264977 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:56.264980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:56.264981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:56.265086 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:56.265363 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:56.265364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:56.265364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:56.265665 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:56.265814 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:56.266090 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 03:21:56.266194 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260927 03:21:56.267371 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:56.276070 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:56.268941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:39815 (local address 127.0.0.1:46349)
0927 03:21:56.268988 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:56.268989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:56.269000 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:21:56.269021 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:56.269022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:56.269030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:56.269094 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:56.269677 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:56.270428 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:56.271394 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:56.271722 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:56.271794 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:56.273153 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:56.273154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:56.273155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:21:56.273169 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:56.274730 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:56.275150 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:56.275151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:56.275152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:56.275219 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:21:56.275456 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:21:56.275457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:21:56.275458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:21:56.275586 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:56.275690 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:56.275813 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 03:21:56.275875 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:21:56.276871 6425 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:56.277359 6425 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:56.281883 6425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f054fa9aa194eb5936a458ccb241267: Generated new TSK 0
I20260927 03:21:56.282069 6425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:56.287601 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6345
W20260927 03:21:56.291783 6330 connection.cc:535] client connection to 127.0.0.1:46349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46349): Connection reset by peer (error 104)
2026-09-27T03:21:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6444](info): set up 2 sockets
Sep 27 03:21:56 dist-test-slave-7j43 krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260927 03:21:57.313162 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:57 dist-test-slave-7j43 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:57.430929 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:57.513396 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:57.513602 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:57.513625 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:57.515012 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:57.515064 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:57.515079 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:57.515101 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:57.515133 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:57.516783 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:57.517040 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:57.517267 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:57.522167 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:57.522193 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:57.522217 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:57.522274 6457 server_base.cc:1034] running on GCE node
I20260927 03:21:57.522621 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:57.522913 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:57.524070 6457 hybrid_clock.cc:648] HybridClock initialized: now 1790479317524054 us; error 29 us; skew 500 ppm
Sep 27 03:21:57 dist-test-slave-7j43 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:57.530467 6457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:57.530962 6457 webserver.cc:460] Webserver started at http://127.0.0.1:40159/ using document root <none> and password file <none>
I20260927 03:21:57.531211 6457 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:57.531286 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:57.531441 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:57.532562 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "4dc44d2531a84add8e02b5fcdbd3c516"
format_stamp: "Formatted at 2026-09-27 03:21:57 on dist-test-slave-7j43"
server_key: "a1eb08feaaf169a73400395385fcc35a"
server_key_iv: "51bbd30f70465547ef1c97c011762673"
server_key_version: "encryptionkey@0"
I20260927 03:21:57.533028 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "4dc44d2531a84add8e02b5fcdbd3c516"
format_stamp: "Formatted at 2026-09-27 03:21:57 on dist-test-slave-7j43"
server_key: "a1eb08feaaf169a73400395385fcc35a"
server_key_iv: "51bbd30f70465547ef1c97c011762673"
server_key_version: "encryptionkey@0"
I20260927 03:21:57.534471 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:57.535355 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:57.535532 6457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:57.535667 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "4dc44d2531a84add8e02b5fcdbd3c516"
format_stamp: "Formatted at 2026-09-27 03:21:57 on dist-test-slave-7j43"
server_key: "a1eb08feaaf169a73400395385fcc35a"
server_key_iv: "51bbd30f70465547ef1c97c011762673"
server_key_version: "encryptionkey@0"
I20260927 03:21:57.535751 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:57.552860 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:57.554869 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:57.555049 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:57.559417 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37985
I20260927 03:21:57.559468 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37985 every 8 connection(s)
I20260927 03:21:57.559859 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:57.560894 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:57.563350 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516: Bootstrap starting.
I20260927 03:21:57.563983 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:57.564307 6525 log.cc:824] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:57.565137 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516: No bootstrap required, opened a new log
I20260927 03:21:57.565994 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6457
I20260927 03:21:57.566149 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:57.566366 17618 external_mini_cluster.cc:1428] Setting key 8bc122d480db438d1e2a1379afd6e970
I20260927 03:21:57.567194 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } }
I20260927 03:21:57.567335 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:57.567358 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc44d2531a84add8e02b5fcdbd3c516, State: Initialized, Role: FOLLOWER
I20260927 03:21:57.567479 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } }
I20260927 03:21:57.567585 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:57.567632 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:57.567683 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:57.568250 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } }
I20260927 03:21:57.568346 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc44d2531a84add8e02b5fcdbd3c516; no voters:
I20260927 03:21:57.568572 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:57.568620 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:57.568779 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc44d2531a84add8e02b5fcdbd3c516, State: Running, Role: LEADER
I20260927 03:21:57.568862 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:57.568915 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } }
I20260927 03:21:57.569303 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc44d2531a84add8e02b5fcdbd3c516. Latest consensus state: current_term: 1 leader_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } } }
I20260927 03:21:57.569465 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:57.569896 6539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:57.570461 6539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:57.570601 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc44d2531a84add8e02b5fcdbd3c516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37985 } } }
I20260927 03:21:57.570715 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:57.572211 6539 catalog_manager.cc:1347] Generated new cluster ID: 78cfa7d52a4a4880ae8ea8f1e763686d
I20260927 03:21:57.572273 6539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:57.572563 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:57.566905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:60149 (local address 127.0.0.1:37985)
0927 03:21:57.567084 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:57.567092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:57.567121 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:21:57.567214 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:57.567220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:57.567268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:57.567408 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:57.567632 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:57.569173 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:57.570329 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:57.570925 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:57.571040 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:57.571802 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:57.571806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:57.571807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:57.572010 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:57.572049 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:57.572063 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:57.572147 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:57.572329 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:21:57.572358 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260927 03:21:57.573506 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:57.578437 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:57.575033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:44271 (local address 127.0.0.1:37985)
0927 03:21:57.575067 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:57.575069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:57.575129 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:21:57.575182 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:57.575182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:57.575193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:57.575261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:57.575379 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:57.576100 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:57.577084 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:21:57.577428 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:21:57.577497 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:21:57.578170 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:57.578171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:57.578172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:57.578176 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:57.578193 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:57.578198 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:57.578257 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:57.578336 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:21:57.578348 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260927 03:21:57.595534 6539 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:57.596338 6539 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:57.601122 6539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc44d2531a84add8e02b5fcdbd3c516: Generated new TSK 0
I20260927 03:21:57.601303 6539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:57.606717 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6457
W20260927 03:21:57.611009 6442 connection.cc:535] client connection to 127.0.0.1:37985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37985): Connection reset by peer (error 104)
2026-09-27T03:21:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:57 dist-test-slave-7j43 krb5kdc[6556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:57 dist-test-slave-7j43 krb5kdc[6556](info): set up 2 sockets
Sep 27 03:21:57 dist-test-slave-7j43 krb5kdc[6556](info): commencing operation
krb5kdc: starting...
W20260927 03:21:58.637290 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:21:58 dist-test-slave-7j43 krb5kdc[6556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:21:58.753338 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:21:58.835085 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:21:58.835289 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:21:58.835314 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:21:58.836469 6569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:21:58.836512 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:21:58.836527 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:21:58.836540 6569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:21:58.836623 6569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:21:58.838233 6569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:21:58.838483 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:21:58.838717 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:21:58.843782 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:58.843886 6569 server_base.cc:1034] running on GCE node
W20260927 03:21:58.843767 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:21:58.843765 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:21:58.844228 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:21:58.844512 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:21:58.845726 6569 hybrid_clock.cc:648] HybridClock initialized: now 1790479318845701 us; error 38 us; skew 500 ppm
Sep 27 03:21:58 dist-test-slave-7j43 krb5kdc[6556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:21:58.848306 6569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:21:58.848814 6569 webserver.cc:460] Webserver started at http://127.0.0.1:39257/ using document root <none> and password file <none>
I20260927 03:21:58.849061 6569 fs_manager.cc:359] Metadata directory not provided
I20260927 03:21:58.849128 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:21:58.849293 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:21:58.850368 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "de4e40b5d6ab4a4e9f3ed3421af94560"
format_stamp: "Formatted at 2026-09-27 03:21:58 on dist-test-slave-7j43"
server_key: "475692e911be91439b9dd5efafd9064c"
server_key_iv: "8d06cc19f4cd9df7b19196492afe2b80"
server_key_version: "encryptionkey@0"
I20260927 03:21:58.850739 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "de4e40b5d6ab4a4e9f3ed3421af94560"
format_stamp: "Formatted at 2026-09-27 03:21:58 on dist-test-slave-7j43"
server_key: "475692e911be91439b9dd5efafd9064c"
server_key_iv: "8d06cc19f4cd9df7b19196492afe2b80"
server_key_version: "encryptionkey@0"
I20260927 03:21:58.852077 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:21:58.852854 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:21:58.853024 6569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:21:58.853130 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "de4e40b5d6ab4a4e9f3ed3421af94560"
format_stamp: "Formatted at 2026-09-27 03:21:58 on dist-test-slave-7j43"
server_key: "475692e911be91439b9dd5efafd9064c"
server_key_iv: "8d06cc19f4cd9df7b19196492afe2b80"
server_key_version: "encryptionkey@0"
I20260927 03:21:58.853230 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:21:58.856658 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:21:58.859265 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:21:58.859423 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:21:58.863646 6569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34403
I20260927 03:21:58.863687 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34403 every 8 connection(s)
I20260927 03:21:58.864061 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:21:58.865047 6637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:21:58.867422 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560: Bootstrap starting.
I20260927 03:21:58.867966 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6569
I20260927 03:21:58.867986 6637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560: Neither blocks nor log segments found. Creating new log.
I20260927 03:21:58.868117 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:21:58.868322 17618 external_mini_cluster.cc:1428] Setting key 6d7cb8c33b94bb69b1b7ffc585f32c66
I20260927 03:21:58.868345 6637 log.cc:824] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560: Log is configured to *not* fsync() on all Append() calls
I20260927 03:21:58.869266 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560: No bootstrap required, opened a new log
I20260927 03:21:58.871026 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:58.868942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:41795 (local address 127.0.0.1:34403)
0927 03:21:58.869132 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:58.869141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:58.869173 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:21:58.869675 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:58.869682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:58.869710 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:58.869861 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:58.870569 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:58.870572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:58.870573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:58.870577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:58.870626 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:58.870637 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:58.870780 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:58.870814 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:21:58.870857 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260927 03:21:58.871963 6637 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260927 03:21:58.872153 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:21:58.872182 6637 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:21:58.872256 6637 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4e40b5d6ab4a4e9f3ed3421af94560, State: Initialized, Role: FOLLOWER
I20260927 03:21:58.872411 6637 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260927 03:21:58.872520 6637 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:21:58.872605 6637 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:21:58.872664 6637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:21:58.873355 6637 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260927 03:21:58.873487 6637 leader_election.cc:302] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4e40b5d6ab4a4e9f3ed3421af94560; no voters:
I20260927 03:21:58.873659 6637 leader_election.cc:288] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:21:58.873842 6648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:21:58.873958 6637 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:21:58.874037 6648 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [term 1 LEADER]: Becoming Leader. State: Replica: de4e40b5d6ab4a4e9f3ed3421af94560, State: Running, Role: LEADER
I20260927 03:21:58.874176 6648 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260927 03:21:58.874511 6648 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4e40b5d6ab4a4e9f3ed3421af94560. Latest consensus state: current_term: 1 leader_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260927 03:21:58.874590 6648 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:58.874519 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e40b5d6ab4a4e9f3ed3421af94560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260927 03:21:58.874841 6655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:21:58.874864 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560 [sys.catalog]: This master's current role is: LEADER
I20260927 03:21:58.875023 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:21:58.873669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:45375 (local address 127.0.0.1:34403)
0927 03:21:58.873722 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:21:58.873724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:21:58.873907 (+ 183us) server_negotiation.cc:407] Connection header received
0927 03:21:58.873969 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:21:58.873970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:21:58.873976 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:21:58.874038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:21:58.874781 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:21:58.874782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:21:58.874782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:21:58.874787 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:21:58.874802 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:21:58.874806 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:21:58.874837 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:21:58.874934 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 03:21:58.874949 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:21:58.875579 6655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:21:58.877421 6655 catalog_manager.cc:1347] Generated new cluster ID: c70265e0ddb24de3b445defc771b3fcc
I20260927 03:21:58.877480 6655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:21:58.888607 6655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:21:58.888818 6655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:21:58.895838 6655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4e40b5d6ab4a4e9f3ed3421af94560: Generated new TSK 0
I20260927 03:21:58.896042 6655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:21:58.902585 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6569
2026-09-27T03:21:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:21:58 dist-test-slave-7j43 krb5kdc[6668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:21:58 dist-test-slave-7j43 krb5kdc[6668](info): set up 2 sockets
Sep 27 03:21:58 dist-test-slave-7j43 krb5kdc[6668](info): commencing operation
krb5kdc: starting...
W20260927 03:21:59.933279 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:00.049523 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43787
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:43787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:00.131857 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:00.132100 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:00.132141 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:00.133327 6681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:00.133396 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:00.133420 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:00.133447 6681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:00.133479 6681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:00.134955 6681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:43787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:00.135229 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:00.135457 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:00.140254 6687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:00.140337 6686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:00.140249 6689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:00.140501 6681 server_base.cc:1034] running on GCE node
I20260927 03:22:00.140766 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:00.141113 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:00.142279 6681 hybrid_clock.cc:648] HybridClock initialized: now 1790479320142263 us; error 45 us; skew 500 ppm
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479320, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:00.144732 6681 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:00.145205 6681 webserver.cc:460] Webserver started at http://127.17.52.190:42551/ using document root <none> and password file <none>
I20260927 03:22:00.145426 6681 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:00.145470 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:00.145610 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:00.146687 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "af31850fee6048be8ec7c31b57082f01"
format_stamp: "Formatted at 2026-09-27 03:22:00 on dist-test-slave-7j43"
server_key: "b510226de40da87567664d6f3567e9ae"
server_key_iv: "3ce5409432594503305ce6d3b8ae2c04"
server_key_version: "encryptionkey@0"
I20260927 03:22:00.147056 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "af31850fee6048be8ec7c31b57082f01"
format_stamp: "Formatted at 2026-09-27 03:22:00 on dist-test-slave-7j43"
server_key: "b510226de40da87567664d6f3567e9ae"
server_key_iv: "3ce5409432594503305ce6d3b8ae2c04"
server_key_version: "encryptionkey@0"
I20260927 03:22:00.148415 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:22:00.149283 6696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:00.149487 6681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:00.149583 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "af31850fee6048be8ec7c31b57082f01"
format_stamp: "Formatted at 2026-09-27 03:22:00 on dist-test-slave-7j43"
server_key: "b510226de40da87567664d6f3567e9ae"
server_key_iv: "3ce5409432594503305ce6d3b8ae2c04"
server_key_version: "encryptionkey@0"
I20260927 03:22:00.149657 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:00.157706 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:00.159762 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:00.159922 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:00.164013 6681 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:43787
I20260927 03:22:00.164048 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:43787 every 8 connection(s)
I20260927 03:22:00.164372 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:00.165194 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:00.167444 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01: Bootstrap starting.
I20260927 03:22:00.168023 6749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:00.168320 6749 log.cc:824] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:00.169073 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01: No bootstrap required, opened a new log
I20260927 03:22:00.170694 6749 raft_consensus.cc:348] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } }
I20260927 03:22:00.170853 6749 raft_consensus.cc:374] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:00.170903 6749 raft_consensus.cc:729] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af31850fee6048be8ec7c31b57082f01, State: Initialized, Role: FOLLOWER
I20260927 03:22:00.171072 6749 consensus_queue.cc:260] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } }
I20260927 03:22:00.171155 6749 raft_consensus.cc:388] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:00.171207 6749 raft_consensus.cc:482] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:00.171259 6749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:00.172060 6749 raft_consensus.cc:504] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } }
I20260927 03:22:00.172188 6749 leader_election.cc:302] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af31850fee6048be8ec7c31b57082f01; no voters:
I20260927 03:22:00.172360 6749 leader_election.cc:288] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:00.172410 6752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:00.172606 6752 raft_consensus.cc:686] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [term 1 LEADER]: Becoming Leader. State: Replica: af31850fee6048be8ec7c31b57082f01, State: Running, Role: LEADER
I20260927 03:22:00.172639 6749 sys_catalog.cc:564] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:00.172744 6752 consensus_queue.cc:237] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } }
I20260927 03:22:00.173113 6753 sys_catalog.cc:455] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af31850fee6048be8ec7c31b57082f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } } }
I20260927 03:22:00.173121 6754 sys_catalog.cc:455] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af31850fee6048be8ec7c31b57082f01. Latest consensus state: current_term: 1 leader_uuid: "af31850fee6048be8ec7c31b57082f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af31850fee6048be8ec7c31b57082f01" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 43787 } } }
I20260927 03:22:00.173278 6754 sys_catalog.cc:458] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:00.173494 6761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:00.173687 6753 sys_catalog.cc:458] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:00.173993 6761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:00.174254 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6681
I20260927 03:22:00.174367 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:00.174525 17618 external_mini_cluster.cc:1428] Setting key 9f3a0847ce27825f4d4c67451f4dc384
I20260927 03:22:00.175582 6761 catalog_manager.cc:1347] Generated new cluster ID: 740ee805064a427a8c76782f227d0d1a
I20260927 03:22:00.175629 6761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:00.189440 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:00.174941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54391 (local address 127.17.52.190:43787)
0927 03:22:00.175107 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:00.175116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:00.175154 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:22:00.175325 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:00.175331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:00.175374 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:00.175583 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:00.176419 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:00.177918 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:00.179057 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:00.180006 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:00.180128 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:00.185419 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:00.185424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:00.185424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:00.185438 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:00.187631 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:00.188100 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:00.188103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:00.188103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:00.188204 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:00.188440 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:00.188441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:00.188442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:00.188773 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:00.188932 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:00.189071 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:22:00.189178 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260927 03:22:00.190438 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:00.191032 6761 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:00.191735 6761 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:00.198650 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:00.192002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60839 (local address 127.17.52.190:43787)
0927 03:22:00.192041 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:00.192043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:00.192150 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:22:00.192230 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:00.192231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:00.192240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:00.192294 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:00.192908 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:00.193746 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:00.194522 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:00.194786 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:00.194874 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:00.196032 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:00.196034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:00.196034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:00.196041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:00.197500 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:00.197865 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:00.197866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:00.197866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:00.197918 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:00.198144 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:00.198146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:00.198146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:00.198253 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:00.198378 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:00.198487 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:22:00.198547 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:22:00.201064 6761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af31850fee6048be8ec7c31b57082f01: Generated new TSK 0
I20260927 03:22:00.201256 6761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:00.205417 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6681
W20260927 03:22:00.209663 6666 connection.cc:535] client connection to 127.17.52.190:43787 recv error: Network error: failed to read from TLS socket (remote: 127.17.52.190:43787): Connection reset by peer (error 104)
2026-09-27T03:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6780](info): set up 2 sockets
Sep 27 03:22:00 dist-test-slave-7j43 krb5kdc[6780](info): commencing operation
krb5kdc: starting...
W20260927 03:22:01.229123 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:01 dist-test-slave-7j43 krb5kdc[6780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:01.346976 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42371
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:01.429234 6793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:01.429468 6793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:01.429494 6793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:01.430655 6793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:01.430704 6793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:01.430721 6793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:01.430737 6793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:01.430750 6793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:01.432435 6793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:42371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:01.432718 6793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:01.432952 6793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:01.437894 6798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:01.437902 6799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:01.437914 6801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:01.437950 6793 server_base.cc:1034] running on GCE node
I20260927 03:22:01.438297 6793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:01.438598 6793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:01.439764 6793 hybrid_clock.cc:648] HybridClock initialized: now 1790479321439748 us; error 45 us; skew 500 ppm
Sep 27 03:22:01 dist-test-slave-7j43 krb5kdc[6780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479321, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:01.442284 6793 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:01.442745 6793 webserver.cc:460] Webserver started at http://127.17.52.190:34675/ using document root <none> and password file <none>
I20260927 03:22:01.442987 6793 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:01.443055 6793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:01.443207 6793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:01.444296 6793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "aa0365fbca7740728fe14ab22f8acbe0"
format_stamp: "Formatted at 2026-09-27 03:22:01 on dist-test-slave-7j43"
server_key: "4e34c679c2ce199270ae76a2deb2e25b"
server_key_iv: "a7172071f3451569c15e8d4980254565"
server_key_version: "encryptionkey@0"
I20260927 03:22:01.444726 6793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "aa0365fbca7740728fe14ab22f8acbe0"
format_stamp: "Formatted at 2026-09-27 03:22:01 on dist-test-slave-7j43"
server_key: "4e34c679c2ce199270ae76a2deb2e25b"
server_key_iv: "a7172071f3451569c15e8d4980254565"
server_key_version: "encryptionkey@0"
I20260927 03:22:01.446046 6793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:01.446789 6808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:01.446959 6793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:01.447058 6793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "aa0365fbca7740728fe14ab22f8acbe0"
format_stamp: "Formatted at 2026-09-27 03:22:01 on dist-test-slave-7j43"
server_key: "4e34c679c2ce199270ae76a2deb2e25b"
server_key_iv: "a7172071f3451569c15e8d4980254565"
server_key_version: "encryptionkey@0"
I20260927 03:22:01.447141 6793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:01.463428 6793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:01.465736 6793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:01.465899 6793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:01.470008 6793 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:42371
I20260927 03:22:01.470042 6860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:42371 every 8 connection(s)
I20260927 03:22:01.470436 6793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:01.471377 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:01.472074 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6793
I20260927 03:22:01.472211 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:01.472425 17618 external_mini_cluster.cc:1428] Setting key 641eec53e8e433b85a845c88f498c871
I20260927 03:22:01.474615 6861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: Bootstrap starting.
I20260927 03:22:01.475339 6861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:01.475682 6861 log.cc:824] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:01.476447 6861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: No bootstrap required, opened a new log
I20260927 03:22:01.478406 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:01.473054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35945 (local address 127.17.52.190:42371)
0927 03:22:01.473225 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:01.473236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:01.473263 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:22:01.473664 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:01.473679 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:01.473727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:01.473842 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:01.474116 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:01.475086 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:01.476241 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:01.476746 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:01.476827 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:01.477581 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:01.477585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:01.477585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:01.477756 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:01.477788 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:01.477798 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:01.477865 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:01.478012 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:22:01.478064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260927 03:22:01.478514 6861 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } }
I20260927 03:22:01.478703 6861 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:01.478739 6861 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0365fbca7740728fe14ab22f8acbe0, State: Initialized, Role: FOLLOWER
I20260927 03:22:01.478911 6861 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } }
I20260927 03:22:01.479014 6861 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:01.479081 6861 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:01.479139 6861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:01.479429 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:01.479856 6861 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } }
I20260927 03:22:01.479987 6861 leader_election.cc:302] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0365fbca7740728fe14ab22f8acbe0; no voters:
I20260927 03:22:01.480175 6861 leader_election.cc:288] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:01.480218 6866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:01.480423 6866 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [term 1 LEADER]: Becoming Leader. State: Replica: aa0365fbca7740728fe14ab22f8acbe0, State: Running, Role: LEADER
I20260927 03:22:01.480505 6861 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:01.480602 6866 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } }
I20260927 03:22:01.480959 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0365fbca7740728fe14ab22f8acbe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } } }
I20260927 03:22:01.481101 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:01.480952 6866 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0365fbca7740728fe14ab22f8acbe0. Latest consensus state: current_term: 1 leader_uuid: "aa0365fbca7740728fe14ab22f8acbe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0365fbca7740728fe14ab22f8acbe0" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 42371 } } }
I20260927 03:22:01.481422 6866 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0 [sys.catalog]: This master's current role is: LEADER
W20260927 03:22:01.482503 6887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:01.482560 6887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:01.482631 6878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:01.482862 6878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:01.484678 6878 catalog_manager.cc:1347] Generated new cluster ID: c274e690f7dd429d99529f1922b11a99
I20260927 03:22:01.484735 6878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:01.485167 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:01.480953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33189 (local address 127.17.52.190:42371)
0927 03:22:01.480986 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:01.480988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:01.481241 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:22:01.481342 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:01.481343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:01.481353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:01.481382 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:01.481611 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:01.482490 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:01.483657 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:01.483983 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:01.484055 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:01.484864 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:01.484866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:01.484866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:01.484871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:01.484886 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:01.484891 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:01.484958 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:01.485081 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 03:22:01.485092 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:22:01.492370 6878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:01.493027 6878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:01.498651 6878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0365fbca7740728fe14ab22f8acbe0: Generated new TSK 0
I20260927 03:22:01.498868 6878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:01.501972 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6793
W20260927 03:22:01.506503 6778 connection.cc:535] client connection to 127.17.52.190:42371 recv error: Network error: failed to read from TLS socket (remote: 127.17.52.190:42371): Connection reset by peer (error 104)
2026-09-27T03:22:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:01 dist-test-slave-7j43 krb5kdc[6892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:01 dist-test-slave-7j43 krb5kdc[6892](info): set up 2 sockets
Sep 27 03:22:01 dist-test-slave-7j43 krb5kdc[6892](info): commencing operation
krb5kdc: starting...
W20260927 03:22:02.565339 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:02 dist-test-slave-7j43 krb5kdc[6892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:02.679520 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44271
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:40187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:02.759827 6905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:02.760070 6905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:02.760108 6905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:02.761328 6905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:02.761399 6905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:02.761425 6905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:02.761456 6905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:02.761483 6905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:02.762943 6905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:40187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:44271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:02.763209 6905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:02.763432 6905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:02.768005 6910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:02.768013 6911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:02.768013 6913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:02.768242 6905 server_base.cc:1034] running on GCE node
I20260927 03:22:02.768486 6905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:02.768860 6905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:02.770011 6905 hybrid_clock.cc:648] HybridClock initialized: now 1790479322769998 us; error 28 us; skew 500 ppm
Sep 27 03:22:02 dist-test-slave-7j43 krb5kdc[6892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479322, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:02.775944 6905 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:02.776413 6905 webserver.cc:460] Webserver started at http://127.17.52.190:36039/ using document root <none> and password file <none>
I20260927 03:22:02.776684 6905 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:02.776731 6905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:02.776866 6905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:02.777911 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "66180fa0c1d649bab862832505a948d8"
format_stamp: "Formatted at 2026-09-27 03:22:02 on dist-test-slave-7j43"
server_key: "bede762e3716dbafff89d91430e083b6"
server_key_iv: "6a143504d03989d9bf87fd14cb3d207e"
server_key_version: "encryptionkey@0"
I20260927 03:22:02.778265 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "66180fa0c1d649bab862832505a948d8"
format_stamp: "Formatted at 2026-09-27 03:22:02 on dist-test-slave-7j43"
server_key: "bede762e3716dbafff89d91430e083b6"
server_key_iv: "6a143504d03989d9bf87fd14cb3d207e"
server_key_version: "encryptionkey@0"
I20260927 03:22:02.779578 6905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:22:02.780303 6920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:02.780627 6905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:02.780737 6905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "66180fa0c1d649bab862832505a948d8"
format_stamp: "Formatted at 2026-09-27 03:22:02 on dist-test-slave-7j43"
server_key: "bede762e3716dbafff89d91430e083b6"
server_key_iv: "6a143504d03989d9bf87fd14cb3d207e"
server_key_version: "encryptionkey@0"
I20260927 03:22:02.780839 6905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:02.784183 6905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:02.786604 6905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:02.786806 6905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:02.790901 6905 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:44271
I20260927 03:22:02.790931 6972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:44271 every 8 connection(s)
I20260927 03:22:02.791280 6905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:02.792243 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:02.793844 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 6905
I20260927 03:22:02.793972 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:02.794157 17618 external_mini_cluster.cc:1428] Setting key 94f45c041d3cf185d5a3f33e1acaa99c
I20260927 03:22:02.794839 6973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: Bootstrap starting.
I20260927 03:22:02.795554 6973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:02.795889 6973 log.cc:824] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:02.796708 6973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: No bootstrap required, opened a new log
I20260927 03:22:02.796936 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:02.794752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40907 (local address 127.17.52.190:44271)
0927 03:22:02.794946 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:02.794956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:02.794991 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:22:02.795580 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:02.795589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:02.795624 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:02.795785 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:02.796416 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:02.796420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:02.796420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:02.796424 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:02.796475 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:02.796486 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:02.796578 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:02.796651 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:22:02.796702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260927 03:22:02.798027 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:02.799003 6973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } }
I20260927 03:22:02.799178 6973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:02.799223 6973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66180fa0c1d649bab862832505a948d8, State: Initialized, Role: FOLLOWER
I20260927 03:22:02.799357 6973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } }
I20260927 03:22:02.799468 6973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:02.799515 6973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:02.799564 6973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:02.800087 6973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } }
I20260927 03:22:02.800216 6973 leader_election.cc:302] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66180fa0c1d649bab862832505a948d8; no voters:
I20260927 03:22:02.800402 6973 leader_election.cc:288] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:02.800426 6983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:02.800717 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:02.799345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34559 (local address 127.17.52.190:44271)
0927 03:22:02.799382 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:02.799384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:02.799530 (+ 146us) server_negotiation.cc:407] Connection header received
0927 03:22:02.799625 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:02.799626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:02.799630 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:02.799684 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:02.800270 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:02.800271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:02.800272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:02.800276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:02.800290 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:02.800294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:02.800349 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:02.800398 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 03:22:02.800413 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:22:02.800734 6983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [term 1 LEADER]: Becoming Leader. State: Replica: 66180fa0c1d649bab862832505a948d8, State: Running, Role: LEADER
I20260927 03:22:02.800830 6983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } }
I20260927 03:22:02.800856 6973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:02.801144 6984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66180fa0c1d649bab862832505a948d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } } }
I20260927 03:22:02.801422 6984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:02.801141 6985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66180fa0c1d649bab862832505a948d8. Latest consensus state: current_term: 1 leader_uuid: "66180fa0c1d649bab862832505a948d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66180fa0c1d649bab862832505a948d8" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 44271 } } }
I20260927 03:22:02.801816 6985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8 [sys.catalog]: This master's current role is: LEADER
W20260927 03:22:02.802616 6999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:02.802659 6999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:02.802721 6992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:02.802886 6992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:02.804280 6992 catalog_manager.cc:1347] Generated new cluster ID: 80b4c3d16bab4721868fa49af68d886e
I20260927 03:22:02.804320 6992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:02.818809 6992 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:02.819072 6992 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:02.824726 6992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66180fa0c1d649bab862832505a948d8: Generated new TSK 0
I20260927 03:22:02.824942 6992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:02.831473 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 6905
2026-09-27T03:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:02 dist-test-slave-7j43 krb5kdc[7004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:02 dist-test-slave-7j43 krb5kdc[7004](info): set up 2 sockets
Sep 27 03:22:02 dist-test-slave-7j43 krb5kdc[7004](info): commencing operation
krb5kdc: starting...
W20260927 03:22:03.869230 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:03 dist-test-slave-7j43 krb5kdc[7004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:03.983186 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33425
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:04.064227 7017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:04.064436 7017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:04.064459 7017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:04.065588 7017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:04.065632 7017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:04.065654 7017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:04.065667 7017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:04.065680 7017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:04.067081 7017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:04.067265 7017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:04.067510 7017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:04.072881 7025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:04.072916 7022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:04.073136 7017 server_base.cc:1034] running on GCE node
W20260927 03:22:04.073272 7023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:04.073559 7017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:04.073855 7017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:04.075032 7017 hybrid_clock.cc:648] HybridClock initialized: now 1790479324075004 us; error 37 us; skew 500 ppm
Sep 27 03:22:04 dist-test-slave-7j43 krb5kdc[7004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479324, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:04.077553 7017 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:04.078035 7017 webserver.cc:460] Webserver started at http://127.17.52.190:35985/ using document root <none> and password file <none>
I20260927 03:22:04.078270 7017 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:04.078339 7017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:04.078492 7017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:04.079537 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "e0182cb7638d4b9aa870925cd62ac543"
format_stamp: "Formatted at 2026-09-27 03:22:04 on dist-test-slave-7j43"
server_key: "14db00293590e64aca23d455e5f5d7be"
server_key_iv: "869101436fab250cf9b1f54808100309"
server_key_version: "encryptionkey@0"
I20260927 03:22:04.079912 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "e0182cb7638d4b9aa870925cd62ac543"
format_stamp: "Formatted at 2026-09-27 03:22:04 on dist-test-slave-7j43"
server_key: "14db00293590e64aca23d455e5f5d7be"
server_key_iv: "869101436fab250cf9b1f54808100309"
server_key_version: "encryptionkey@0"
I20260927 03:22:04.081298 7017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:04.081990 7032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:04.082155 7017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:04.082267 7017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "e0182cb7638d4b9aa870925cd62ac543"
format_stamp: "Formatted at 2026-09-27 03:22:04 on dist-test-slave-7j43"
server_key: "14db00293590e64aca23d455e5f5d7be"
server_key_iv: "869101436fab250cf9b1f54808100309"
server_key_version: "encryptionkey@0"
I20260927 03:22:04.082340 7017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:04.091270 7017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:04.093279 7017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:04.093474 7017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:04.097898 7017 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33425
I20260927 03:22:04.097954 7084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33425 every 8 connection(s)
I20260927 03:22:04.098313 7017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:04.099112 7085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:04.101320 7085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543: Bootstrap starting.
I20260927 03:22:04.101837 7085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:04.102066 7085 log.cc:824] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:04.102702 7085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543: No bootstrap required, opened a new log
I20260927 03:22:04.104194 7085 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } }
I20260927 03:22:04.104324 7085 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:04.104355 7085 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0182cb7638d4b9aa870925cd62ac543, State: Initialized, Role: FOLLOWER
I20260927 03:22:04.104480 7085 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } }
I20260927 03:22:04.104557 7085 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:04.104640 7085 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:04.104691 7085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:04.105511 7085 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } }
I20260927 03:22:04.105594 7085 leader_election.cc:302] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0182cb7638d4b9aa870925cd62ac543; no voters:
I20260927 03:22:04.105762 7085 leader_election.cc:288] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:04.105829 7088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:04.105986 7088 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [term 1 LEADER]: Becoming Leader. State: Replica: e0182cb7638d4b9aa870925cd62ac543, State: Running, Role: LEADER
I20260927 03:22:04.106005 7085 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:04.106091 7088 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } }
I20260927 03:22:04.106472 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0182cb7638d4b9aa870925cd62ac543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } } }
I20260927 03:22:04.106568 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:04.106691 7090 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0182cb7638d4b9aa870925cd62ac543. Latest consensus state: current_term: 1 leader_uuid: "e0182cb7638d4b9aa870925cd62ac543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182cb7638d4b9aa870925cd62ac543" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33425 } } }
I20260927 03:22:04.107033 7090 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:04.107162 7097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:04.107581 7097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:04.108021 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7017
I20260927 03:22:04.108121 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:04.108315 17618 external_mini_cluster.cc:1428] Setting key 3ef12a031fbacc60e009fe7fcfdffd94
I20260927 03:22:04.109522 7097 catalog_manager.cc:1347] Generated new cluster ID: 1be97d6129b444c2834f1505426fe20f
I20260927 03:22:04.109578 7097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:22:04 dist-test-slave-7j43 krb5kdc[7004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:04.119796 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:04.108793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37289 (local address 127.17.52.190:33425)
0927 03:22:04.108992 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:04.109003 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:04.109078 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:22:04.109244 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:04.109251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:04.109307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:04.110226 (+ 919us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:04.110313 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:04.111634 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:04.112670 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:04.113380 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:04.113501 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:04.116245 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:04.116249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:04.116249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:04.116261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:04.118149 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:04.118543 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:04.118546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:04.118547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:04.118647 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:04.118867 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:04.118868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:04.118868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:04.119171 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:04.119302 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:04.119424 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:22:04.119553 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260927 03:22:04.120481 7097 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:04.120995 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:04.121156 7097 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:04.129129 7097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0182cb7638d4b9aa870925cd62ac543: Generated new TSK 0
I20260927 03:22:04.129307 7097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:04.131595 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:04.122381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47671 (local address 127.17.52.190:33425)
0927 03:22:04.122413 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:04.122414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:04.124780 (+ 2366us) server_negotiation.cc:407] Connection header received
0927 03:22:04.124839 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:04.124840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:04.124850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:04.124904 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:04.125470 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:04.126185 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:04.127218 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:04.127911 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:04.127973 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:04.129349 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:04.129350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:04.129351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:04.129357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:04.130609 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:04.130978 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:04.130979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:04.130979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:04.131025 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:04.131203 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:04.131204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:04.131204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:04.131292 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:04.131364 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:04.131488 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 03:22:04.131543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:22:04.133399 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7017
2026-09-27T03:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:04 dist-test-slave-7j43 krb5kdc[7116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:04 dist-test-slave-7j43 krb5kdc[7116](info): set up 2 sockets
Sep 27 03:22:04 dist-test-slave-7j43 krb5kdc[7116](info): commencing operation
krb5kdc: starting...
W20260927 03:22:05.165262 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:05 dist-test-slave-7j43 krb5kdc[7116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:05.283779 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38049
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:05.363593 7129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:05.363830 7129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:05.363865 7129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:05.365053 7129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:05.365116 7129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:05.365139 7129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:05.365167 7129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:05.365199 7129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:05.366638 7129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:38049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:05.366902 7129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:05.367138 7129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:05.371943 7135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:05.371953 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:05.371968 7134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:05.372164 7129 server_base.cc:1034] running on GCE node
I20260927 03:22:05.372408 7129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:05.372768 7129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:05.373935 7129 hybrid_clock.cc:648] HybridClock initialized: now 1790479325373931 us; error 38 us; skew 500 ppm
Sep 27 03:22:05 dist-test-slave-7j43 krb5kdc[7116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479325, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:05.376351 7129 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:05.376793 7129 webserver.cc:460] Webserver started at http://127.17.52.190:38283/ using document root <none> and password file <none>
I20260927 03:22:05.376976 7129 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:05.377014 7129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:05.377103 7129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:05.378149 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "9427722338c24833a6f406514f76ac0b"
format_stamp: "Formatted at 2026-09-27 03:22:05 on dist-test-slave-7j43"
server_key: "ac8fd5e0cf1f76d1c1337b054ae7151d"
server_key_iv: "417e5ff47aa978f2f86f36dcbf1ecf28"
server_key_version: "encryptionkey@0"
I20260927 03:22:05.378458 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "9427722338c24833a6f406514f76ac0b"
format_stamp: "Formatted at 2026-09-27 03:22:05 on dist-test-slave-7j43"
server_key: "ac8fd5e0cf1f76d1c1337b054ae7151d"
server_key_iv: "417e5ff47aa978f2f86f36dcbf1ecf28"
server_key_version: "encryptionkey@0"
I20260927 03:22:05.379670 7129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:05.380297 7144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:05.380488 7129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:05.380609 7129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "9427722338c24833a6f406514f76ac0b"
format_stamp: "Formatted at 2026-09-27 03:22:05 on dist-test-slave-7j43"
server_key: "ac8fd5e0cf1f76d1c1337b054ae7151d"
server_key_iv: "417e5ff47aa978f2f86f36dcbf1ecf28"
server_key_version: "encryptionkey@0"
I20260927 03:22:05.380676 7129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:05.400357 7129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:05.402420 7129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:05.402532 7129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:05.406322 7129 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:38049
I20260927 03:22:05.406375 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:38049 every 8 connection(s)
I20260927 03:22:05.406738 7129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:05.407610 7197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:05.408658 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7129
I20260927 03:22:05.408754 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:05.408954 17618 external_mini_cluster.cc:1428] Setting key 86a5ffcae5355cfbeb19512f60cd3f37
I20260927 03:22:05.411108 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b: Bootstrap starting.
I20260927 03:22:05.411818 7197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:05.412120 7197 log.cc:824] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:05.412907 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b: No bootstrap required, opened a new log
I20260927 03:22:05.414973 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:05.409479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41213 (local address 127.17.52.190:38049)
0927 03:22:05.409673 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:05.409682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:05.409715 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:22:05.410242 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:05.410251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:05.410316 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:05.410469 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:05.410631 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:05.412021 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:05.412872 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:05.413467 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:05.413583 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:05.414213 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:05.414219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:05.414219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:05.414425 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:05.414481 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:05.414498 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:05.414577 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:05.414634 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 03:22:05.414713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260927 03:22:05.415318 7197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } }
I20260927 03:22:05.415447 7197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:05.415468 7197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9427722338c24833a6f406514f76ac0b, State: Initialized, Role: FOLLOWER
I20260927 03:22:05.415655 7197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } }
I20260927 03:22:05.415761 7197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:05.415796 7197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:05.415844 7197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:05.416090 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:05.416477 7197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } }
I20260927 03:22:05.416620 7197 leader_election.cc:302] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9427722338c24833a6f406514f76ac0b; no voters:
I20260927 03:22:05.416849 7197 leader_election.cc:288] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:05.416877 7202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:05.416986 7202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [term 1 LEADER]: Becoming Leader. State: Replica: 9427722338c24833a6f406514f76ac0b, State: Running, Role: LEADER
I20260927 03:22:05.417150 7197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:05.417135 7202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } }
I20260927 03:22:05.417582 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9427722338c24833a6f406514f76ac0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } } }
I20260927 03:22:05.417728 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:05.417943 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9427722338c24833a6f406514f76ac0b. Latest consensus state: current_term: 1 leader_uuid: "9427722338c24833a6f406514f76ac0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427722338c24833a6f406514f76ac0b" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 38049 } } }
I20260927 03:22:05.418051 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:05.418458 7216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:05.419132 7216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:05.420922 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:05.417597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58515 (local address 127.17.52.190:38049)
0927 03:22:05.417633 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:05.417635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:05.417758 (+ 123us) server_negotiation.cc:407] Connection header received
0927 03:22:05.417867 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:05.417868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:05.417878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:05.417908 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:05.418112 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:05.418835 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:05.419614 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:05.419861 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:05.419925 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:05.420675 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:05.420676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:05.420676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:05.420682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:05.420697 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:05.420701 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:05.420761 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:05.420829 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:22:05.420852 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:05.421075 7216 catalog_manager.cc:1347] Generated new cluster ID: 5f125751fe1944fabbb828f08f5371c6
I20260927 03:22:05.421146 7216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:05.451915 7216 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:05.452510 7216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:05.455549 7216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9427722338c24833a6f406514f76ac0b: Generated new TSK 0
I20260927 03:22:05.455766 7216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:05.458483 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7129
2026-09-27T03:22:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:05 dist-test-slave-7j43 krb5kdc[7228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:05 dist-test-slave-7j43 krb5kdc[7228](info): set up 2 sockets
Sep 27 03:22:05 dist-test-slave-7j43 krb5kdc[7228](info): commencing operation
krb5kdc: starting...
W20260927 03:22:06.493278 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:06 dist-test-slave-7j43 krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:06.610833 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33769
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:06.690058 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:06.690263 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:06.690285 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:06.691448 7241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:06.691489 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:06.691504 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:06.691517 7241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:06.691529 7241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:06.692992 7241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:06.693238 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:06.693475 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:06.698223 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:06.698269 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:06.698287 7241 server_base.cc:1034] running on GCE node
W20260927 03:22:06.698233 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:06.698616 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:06.698887 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:06.700027 7241 hybrid_clock.cc:648] HybridClock initialized: now 1790479326700018 us; error 30 us; skew 500 ppm
Sep 27 03:22:06 dist-test-slave-7j43 krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479326, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:06.702481 7241 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:06.702934 7241 webserver.cc:460] Webserver started at http://127.17.52.190:44633/ using document root <none> and password file <none>
I20260927 03:22:06.703148 7241 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:06.703193 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:06.703346 7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:06.704346 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "be914d01c0a345dc94c571559bc233d6"
format_stamp: "Formatted at 2026-09-27 03:22:06 on dist-test-slave-7j43"
server_key: "60ce61e5b63631123b26bf27e70b3c90"
server_key_iv: "ee8ac148d361cfb45a8c84b5308428ae"
server_key_version: "encryptionkey@0"
I20260927 03:22:06.704726 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "be914d01c0a345dc94c571559bc233d6"
format_stamp: "Formatted at 2026-09-27 03:22:06 on dist-test-slave-7j43"
server_key: "60ce61e5b63631123b26bf27e70b3c90"
server_key_iv: "ee8ac148d361cfb45a8c84b5308428ae"
server_key_version: "encryptionkey@0"
I20260927 03:22:06.706037 7241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:06.706809 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:06.707002 7241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:06.707091 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "be914d01c0a345dc94c571559bc233d6"
format_stamp: "Formatted at 2026-09-27 03:22:06 on dist-test-slave-7j43"
server_key: "60ce61e5b63631123b26bf27e70b3c90"
server_key_iv: "ee8ac148d361cfb45a8c84b5308428ae"
server_key_version: "encryptionkey@0"
I20260927 03:22:06.707162 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:06.710479 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:06.712756 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:06.712908 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:06.716913 7241 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:33769
I20260927 03:22:06.716961 7308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:33769 every 8 connection(s)
I20260927 03:22:06.717269 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:06.718236 7309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:06.720367 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6: Bootstrap starting.
I20260927 03:22:06.720989 7309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:06.721232 7309 log.cc:824] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:06.721858 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6: No bootstrap required, opened a new log
I20260927 03:22:06.723380 7309 raft_consensus.cc:348] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } }
I20260927 03:22:06.723506 7309 raft_consensus.cc:374] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:06.723528 7309 raft_consensus.cc:729] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be914d01c0a345dc94c571559bc233d6, State: Initialized, Role: FOLLOWER
I20260927 03:22:06.723623 7309 consensus_queue.cc:260] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } }
I20260927 03:22:06.723677 7309 raft_consensus.cc:388] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:06.723698 7309 raft_consensus.cc:482] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:06.723728 7309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:06.724448 7309 raft_consensus.cc:504] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } }
I20260927 03:22:06.724529 7309 leader_election.cc:302] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be914d01c0a345dc94c571559bc233d6; no voters:
I20260927 03:22:06.724725 7309 leader_election.cc:288] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:06.724776 7312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:06.724944 7312 raft_consensus.cc:686] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [term 1 LEADER]: Becoming Leader. State: Replica: be914d01c0a345dc94c571559bc233d6, State: Running, Role: LEADER
I20260927 03:22:06.725057 7312 consensus_queue.cc:237] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } }
I20260927 03:22:06.725162 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7241
I20260927 03:22:06.725271 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:06.725430 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be914d01c0a345dc94c571559bc233d6. Latest consensus state: current_term: 1 leader_uuid: "be914d01c0a345dc94c571559bc233d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } } }
I20260927 03:22:06.725484 17618 external_mini_cluster.cc:1428] Setting key 4ae44bcf9c1c1b38110c950dcd2116ba
I20260927 03:22:06.725419 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be914d01c0a345dc94c571559bc233d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be914d01c0a345dc94c571559bc233d6" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 33769 } } }
I20260927 03:22:06.725633 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:06.725659 7309 sys_catalog.cc:564] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:06.725555 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:06.726054 7316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:06.726586 7316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:06.728102 7316 catalog_manager.cc:1347] Generated new cluster ID: de50ed86e2df4c789045b91fcc68422f
I20260927 03:22:06.728154 7316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:06.728403 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:06.726026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42619 (local address 127.17.52.190:33769)
0927 03:22:06.726210 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:06.726220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:06.726411 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:22:06.726635 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:06.726642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:06.726686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:06.726817 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:06.727920 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:06.727923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:06.727924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:06.727929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:06.727972 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:06.727987 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:06.728054 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:06.728175 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:22:06.728226 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260927 03:22:06.729418 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:06.732053 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:06.730917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47827 (local address 127.17.52.190:33769)
0927 03:22:06.730951 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:06.730952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:06.731069 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:22:06.731161 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:06.731162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:06.731166 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:06.731194 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:06.731821 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:06.731822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:06.731823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:22:06.731827 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:06.731840 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:22:06.731844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:06.731875 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:06.731980 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:22:06.731992 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:22:06.751634 7316 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:06.751834 7316 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:06.753665 7316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be914d01c0a345dc94c571559bc233d6: Generated new TSK 0
I20260927 03:22:06.753842 7316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:06.762084 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7241
2026-09-27T03:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1303 ms)
[----------] 18 tests from AuthTokenIssuingTest (23575 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:06 dist-test-slave-7j43 krb5kdc[7340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:06 dist-test-slave-7j43 krb5kdc[7340](info): set up 2 sockets
Sep 27 03:22:06 dist-test-slave-7j43 krb5kdc[7340](info): commencing operation
krb5kdc: starting...
W20260927 03:22:07.857326 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:07 dist-test-slave-7j43 krb5kdc[7340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:07.968102 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:35087
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:08.047780 7353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:08.048010 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:08.048048 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:08.048086 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:08.049232 7353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:08.049289 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:08.049315 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:08.049345 7353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:08.049367 7353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:08.050817 7353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:35087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:08.051081 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:08.051301 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:08.056143 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.056185 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.056207 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:08.056231 7353 server_base.cc:1034] running on GCE node
I20260927 03:22:08.056545 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:08.056896 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:08.058084 7353 hybrid_clock.cc:648] HybridClock initialized: now 1790479328058075 us; error 34 us; skew 500 ppm
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:08.063733 7353 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:08.064195 7353 webserver.cc:460] Webserver started at http://127.17.52.190:41587/ using document root <none> and password file <none>
I20260927 03:22:08.064422 7353 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:08.064491 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:08.064664 7353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:08.065690 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "6200e0ae8d3e4a7892db9bc23beb195f"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "20690e4076631e06bd8a36c58cfb7043"
server_key_iv: "4e673b790863ad344b9d1e90b7e61c9e"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.066051 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "6200e0ae8d3e4a7892db9bc23beb195f"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "20690e4076631e06bd8a36c58cfb7043"
server_key_iv: "4e673b790863ad344b9d1e90b7e61c9e"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.067436 7353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:22:08.068235 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:08.068475 7353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:08.068611 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "6200e0ae8d3e4a7892db9bc23beb195f"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "20690e4076631e06bd8a36c58cfb7043"
server_key_iv: "4e673b790863ad344b9d1e90b7e61c9e"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.068687 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:08.078150 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:08.079890 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:08.080051 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:08.084138 7353 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:35087
I20260927 03:22:08.084175 7420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:35087 every 8 connection(s)
I20260927 03:22:08.084524 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:08.085508 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:08.086941 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:08.092715 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7353
I20260927 03:22:08.092834 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:08.093012 17618 external_mini_cluster.cc:1428] Setting key 0a43246a5c49342c97a01cefa6d15a69
I20260927 03:22:08.096206 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.087542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:35087 (local address 127.0.0.1:45380)
0927 03:22:08.087703 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.087742 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.088206 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.088479 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.088481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.088773 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.088929 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.088934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.089952 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.089953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.090638 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.090640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.091097 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.091098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.091150 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.091592 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.091600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.093119 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.094760 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.094766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.094767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.094953 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.095201 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.095202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.095202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.095287 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.095642 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.095647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.095703 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.095808 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.096038 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":101,"thread_start_us":89,"threads_started":1}
I20260927 03:22:08.096210 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.087594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.17.52.190:35087)
0927 03:22:08.087730 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.087746 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.087871 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:22:08.088341 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.088343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.088365 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.088455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.089033 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.089860 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.090716 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.091006 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.091084 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.093217 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.093224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.093224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.093230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.094661 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.095046 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.095049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.095049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.095125 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.095373 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.095374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.095375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.095570 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.095661 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.095849 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.095976 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260927 03:22:08.097069 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:08.097427 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.097305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52136)
0927 03:22:08.097357 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.097397 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:22:08.097597 7369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
I20260927 03:22:08.098030 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.097940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52142)
0927 03:22:08.097991 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.098014 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 03:22:08.098148 7421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:42465: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.17.52.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
I20260927 03:22:08.115339 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:42465
--webserver_interface=127.17.52.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:22:08.147368 7421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } attempt: 1
I20260927 03:22:08.147814 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.147683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52146)
0927 03:22:08.147750 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.147786 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 03:22:08.148154 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.148061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52152)
0927 03:22:08.148107 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.148130 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:22:08.148236 7421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:42465: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
W20260927 03:22:08.195739 7428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:08.195936 7428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:08.195953 7428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:08.195973 7428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:08.197181 7428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:08.197221 7428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:08.197235 7428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:08.197268 7428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:08.197297 7428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:08.198729 7428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.52.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:08.198936 7428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:08.199143 7428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:08.203768 7433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.203826 7436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.203766 7434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:08.203981 7428 server_base.cc:1034] running on GCE node
I20260927 03:22:08.204241 7428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:08.204511 7428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:08.205649 7428 hybrid_clock.cc:648] HybridClock initialized: now 1790479328205626 us; error 44 us; skew 500 ppm
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:08.208068 7428 init.cc:377] Logged in from keytab as kudu/127.17.52.189@KRBTEST.COM (short username kudu)
I20260927 03:22:08.208513 7428 webserver.cc:460] Webserver started at http://127.17.52.189:34965/ using document root <none> and password file <none>
I20260927 03:22:08.208773 7428 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:08.208818 7428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:08.208961 7428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:08.209995 7428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data/instance:
uuid: "d668143132e843a6960eb60f71fb6072"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "a17909ccaa5b5bdaa306b67c8fea96ae"
server_key_iv: "dd3e70044db6f6405586d9cbe3bfa3d3"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.210341 7428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal/instance:
uuid: "d668143132e843a6960eb60f71fb6072"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "a17909ccaa5b5bdaa306b67c8fea96ae"
server_key_iv: "dd3e70044db6f6405586d9cbe3bfa3d3"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.211591 7428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:08.212277 7443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:08.212458 7428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:08.212527 7428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "d668143132e843a6960eb60f71fb6072"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "a17909ccaa5b5bdaa306b67c8fea96ae"
server_key_iv: "dd3e70044db6f6405586d9cbe3bfa3d3"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.212632 7428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:08.216364 7421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } attempt: 2
I20260927 03:22:08.216773 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.216656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52166)
0927 03:22:08.216708 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.216744 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:22:08.217113 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.217034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52182)
0927 03:22:08.217074 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.217096 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:22:08.217209 7421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:42465: Network error: Client connection negotiation failed: client connection to 127.17.52.189:42465: connect: Connection refused (error 111)
I20260927 03:22:08.233685 7428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:08.235826 7428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:08.235931 7428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:08.239934 7428 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.189:42465
I20260927 03:22:08.239964 7495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.189:42465 every 8 connection(s)
I20260927 03:22:08.240348 7428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
I20260927 03:22:08.241303 7496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:08.242812 7496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:08.249763 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7428
I20260927 03:22:08.249873 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-1/wal/instance
I20260927 03:22:08.250068 17618 external_mini_cluster.cc:1428] Setting key 8b5323e6807171f0892c9c56a5c0bc84
I20260927 03:22:08.251873 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.243562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.17.52.190:35087)
0927 03:22:08.243633 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.243635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.243821 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:22:08.244262 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.244263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.244293 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.244356 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.245015 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.245497 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.246424 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.246616 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.246700 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.248745 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.248746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.248746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.248753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.250295 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.250761 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.250762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.250762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.250814 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.251108 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.251110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.251110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.251241 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.251305 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.251652 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.251746 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:22:08.251891 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.243491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:35087 (local address 127.0.0.1:45390)
0927 03:22:08.243691 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.243734 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.244181 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.244431 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.244434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.244738 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.244941 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.244945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.245570 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.245571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.246353 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.246355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.246726 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.246728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.246861 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.247289 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.247297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.248664 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.250364 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.250371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.250373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.250675 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.250881 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.250882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.250882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.251019 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.251314 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.251319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.251484 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.251607 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.251733 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260927 03:22:08.252600 7496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.188@KRBTEST.COM; defaulting to no policy
I20260927 03:22:08.260360 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.252860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52196)
0927 03:22:08.252906 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.252915 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.252952 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.253215 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.253215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.253313 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.253362 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.253363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.254676 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.254676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.255159 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.255160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.255865 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.255866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.255916 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.256293 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.256296 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.257658 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.259422 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.259422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.259423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.259628 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.259818 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.259818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.259818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.259871 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.260073 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.260073 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.260106 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.260154 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.260228 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:22:08.260456 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.252914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.17.52.189:42465)
0927 03:22:08.253069 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.253074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.253095 (+ 21us) server_negotiation.cc:407] Connection header received
0927 03:22:08.253116 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.253118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.253131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.253206 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.253428 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.254584 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.255246 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.255769 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.255844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.257748 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.257755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.257755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.257762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.259350 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.259712 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.259714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.259715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.259763 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.259908 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.259908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.259909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.260023 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.260078 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.260174 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.260303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.188@KRBTEST.COM" created.
I20260927 03:22:08.261114 7496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:08.261452 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.261349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60740)
0927 03:22:08.261394 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.261435 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:22:08.261610 7444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
I20260927 03:22:08.262027 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.261946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60752)
0927 03:22:08.261989 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.262012 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:22:08.262141 7496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:36847: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
I20260927 03:22:08.275806 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:36847
--webserver_interface=127.17.52.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:22:08.308358 7421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } attempt: 3
I20260927 03:22:08.311375 7496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } attempt: 1
I20260927 03:22:08.311781 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.311631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60754)
0927 03:22:08.311704 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.311742 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 03:22:08.312641 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.312491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60760)
0927 03:22:08.312550 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.312603 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 03:22:08.312774 7496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:36847: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:08.316216 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.308874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.17.52.189:42465)
0927 03:22:08.308942 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.308944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.308952 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:08.308979 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.308980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.308991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.309050 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.309489 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.309994 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.311011 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.311227 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.311281 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.313588 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.313589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.313590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.313597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.315088 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.315445 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.315447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.315451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.315498 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.315696 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.315697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.315697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.315809 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.315887 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.316048 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.316126 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:22:08.316222 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.308806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52198)
0927 03:22:08.308873 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.308884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.308928 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.309073 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.309074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.309300 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.309400 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.309403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.310073 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.310074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.310928 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.310930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.311303 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.311305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.311388 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.311964 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.311971 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.313501 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.315170 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.315171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.315171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.315348 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.315565 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.315566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.315566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.315632 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.315880 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.315881 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.315950 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.316023 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.316124 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:22:08.317036 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:08.317378 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.317265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60764)
0927 03:22:08.317314 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.317346 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:22:08.317680 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.317594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60770)
0927 03:22:08.317636 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.317658 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:22:08.317795 7421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:36847: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
I20260927 03:22:08.348953 7421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } attempt: 1
I20260927 03:22:08.349444 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.349285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60786)
0927 03:22:08.349372 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.349414 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 03:22:08.349961 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.349809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60794)
0927 03:22:08.349892 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.349927 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 03:22:08.350091 7421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:36847: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
W20260927 03:22:08.359570 7503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:08.359807 7503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:08.359825 7503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:08.359846 7503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:08.361166 7503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:08.361217 7503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:08.361232 7503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:08.361246 7503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:08.361259 7503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:08.362818 7503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:35087,127.17.52.189:42465,127.17.52.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.52.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:08.363080 7503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:08.363324 7503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:08.368218 7511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.368266 7508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:08.368258 7509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:08.368453 7503 server_base.cc:1034] running on GCE node
I20260927 03:22:08.368773 7503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:08.369028 7503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:08.370216 7503 hybrid_clock.cc:648] HybridClock initialized: now 1790479328370194 us; error 29 us; skew 500 ppm
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:08.372884 7503 init.cc:377] Logged in from keytab as kudu/127.17.52.188@KRBTEST.COM (short username kudu)
I20260927 03:22:08.373347 7503 webserver.cc:460] Webserver started at http://127.17.52.188:35279/ using document root <none> and password file <none>
I20260927 03:22:08.373530 7503 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:08.373570 7503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:08.373675 7503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:08.374671 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data/instance:
uuid: "13108687f45b4b65a798e1c807385416"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "d5f84972114adaf9e058a2af6b6fe41c"
server_key_iv: "46596195c49359280276efca602ae702"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.374966 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal/instance:
uuid: "13108687f45b4b65a798e1c807385416"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "d5f84972114adaf9e058a2af6b6fe41c"
server_key_iv: "46596195c49359280276efca602ae702"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.376262 7503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:08.377072 7518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:08.377242 7503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:08.377329 7503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "13108687f45b4b65a798e1c807385416"
format_stamp: "Formatted at 2026-09-27 03:22:08 on dist-test-slave-7j43"
server_key: "d5f84972114adaf9e058a2af6b6fe41c"
server_key_iv: "46596195c49359280276efca602ae702"
server_key_version: "encryptionkey@0"
I20260927 03:22:08.377386 7503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:08.380920 7496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } attempt: 2
I20260927 03:22:08.381322 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.381190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60806)
0927 03:22:08.381247 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.381286 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:22:08.381735 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.381609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60814)
0927 03:22:08.381681 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.381710 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 03:22:08.381907 7496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:36847: Network error: Client connection negotiation failed: client connection to 127.17.52.188:36847: connect: Connection refused (error 111)
I20260927 03:22:08.397586 7503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:08.399940 7503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:08.400102 7503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:08.404018 7503 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.188:36847
I20260927 03:22:08.404067 7570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.188:36847 every 8 connection(s)
I20260927 03:22:08.404433 7503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
I20260927 03:22:08.405469 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:08.406968 7571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:08.410773 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7503
I20260927 03:22:08.410907 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479066103559-17618-0/minicluster-data/master-2/wal/instance
I20260927 03:22:08.411082 17618 external_mini_cluster.cc:1428] Setting key ffd263583b60f0d3ca7288854145ce36
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:08.415930 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.407653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.17.52.190:35087)
0927 03:22:08.407719 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.407721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.407963 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:22:08.408432 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.408433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.408444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.408503 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.409139 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.409634 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.410547 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.410737 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.410836 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.413126 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.413127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.413128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.413135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.414518 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.414978 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.414979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.414979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.415039 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.415295 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.415296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.415296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.415411 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.415484 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.415761 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.415837 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:22:08.416167 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.407637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:35087 (local address 127.0.0.1:45396)
0927 03:22:08.407849 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.407884 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.408319 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.408622 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.408626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.408874 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.409058 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.409062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.409711 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.409712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.410474 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.410476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.410858 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.410860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.411022 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.411668 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.411676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.413036 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.414619 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.414626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.414628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.414850 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.415107 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.415108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.415108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.415220 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.415477 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.415483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.415616 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.415737 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.415877 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":122,"threads_started":1}
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:08.416761 7571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:08.419734 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.411474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.17.52.188:36847)
0927 03:22:08.411658 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.411663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.411734 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:22:08.411801 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.411803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.411823 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.411925 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.412513 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.413274 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.414034 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.414340 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.414427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.416881 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.416887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.416887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.416894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.418556 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.418925 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.418927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.418927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.418974 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.419163 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.419164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.419164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.419334 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.419464 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.419529 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.419612 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:08.423990 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.417047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.17.52.189:42465)
0927 03:22:08.417097 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.417099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.417158 (+ 59us) server_negotiation.cc:407] Connection header received
0927 03:22:08.417218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.417218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.417228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.417299 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.417590 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.418311 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.419215 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.419516 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.419580 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.421485 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.421486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.421487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.421492 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.422971 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.423333 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.423334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.423334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.423383 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.423576 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.423577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.423577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.423660 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.423702 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.423879 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.423931 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:22:08.424081 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.416983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:42465 (local address 127.0.0.1:52210)
0927 03:22:08.417091 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.417102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.417158 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.417304 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.417304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.417471 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.417516 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.417517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.418407 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.418408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.419139 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.419142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.419597 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.419598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.419674 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.420183 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.420188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.421404 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.423043 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.423044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.423045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.423264 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.423445 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.423446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.423446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.423513 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.423720 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.423721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.423771 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.423864 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.423958 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 03:22:08.424422 7571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:08.425262 7421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } attempt: 2
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:08.428861 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.420503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.17.52.190:35087)
0927 03:22:08.420542 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.420544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.420552 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:08.420589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.420590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.420600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.420672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.421093 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.421712 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.422541 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.422848 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.422924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.425750 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.425751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.425751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.425757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.427189 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.427764 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.427766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.427766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.427827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.428030 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.428031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.428031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.428138 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.428315 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.428646 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.428717 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:08.431752 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.424791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60818)
0927 03:22:08.424868 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.424879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.424927 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.425044 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.425044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.425167 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.425189 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.425190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.426182 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.426183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.426907 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.426909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.427351 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.427351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.427396 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.427716 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.427721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.429018 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.430686 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.430686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.430687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.430921 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.431116 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.431117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.431117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.431174 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.431435 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.431436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.431473 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.431558 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.431646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:22:08.432165 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.424836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.17.52.188:36847)
0927 03:22:08.424883 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.424885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.424912 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:22:08.424979 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.424980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.424989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.425040 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.425414 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.426101 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.426978 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.427267 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.427514 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.429105 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.429106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.429106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.429113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.430600 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.430996 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.430997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.430997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.431047 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.431242 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.431242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.431242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.431358 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.431437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.431581 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.432070 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":374,"server-negotiator.queue_time_us":32}
I20260927 03:22:08.433840 7571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: Bootstrap starting.
I20260927 03:22:08.434118 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.425495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60824)
0927 03:22:08.425535 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.425544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.425577 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.425794 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.425794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.425988 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.426022 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.426023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.426946 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.426947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.427739 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.427742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.428225 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.428226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.428281 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.428665 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.428669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.430096 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.433146 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.433146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.433147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.433312 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.433534 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.433535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.433535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.433590 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.433867 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.433868 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.433911 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.433975 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.434056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:22:08.434500 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.425549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.17.52.188:36847)
0927 03:22:08.425677 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.425679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.425685 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:22:08.425701 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.425702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.425709 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.426082 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.426119 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.426851 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.427841 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.428154 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.428214 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.430523 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.430525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.430525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.430532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.432880 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.433404 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.433405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.433406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.433458 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.433662 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.433663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.433664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.433796 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.433874 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.434323 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.434396 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:08.434693 7571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:08.435002 7571 log.cc:824] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:08.436313 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: Bootstrap starting.
I20260927 03:22:08.437042 7421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:08.437134 7571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: No bootstrap required, opened a new log
I20260927 03:22:08.437361 7421 log.cc:824] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:08.438102 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.430287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.17.52.189:42465)
0927 03:22:08.430320 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.430322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.430330 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:08.430349 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.430350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.430359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.430416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.431032 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.431540 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.432314 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.432662 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.432736 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.435234 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.435235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.435235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.435242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.436793 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.437198 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.437199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.437199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.437268 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.437480 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.437481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.437481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.437608 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.437727 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.437921 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.437995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:22:08.438249 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: No bootstrap required, opened a new log
I20260927 03:22:08.439343 7571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.439502 7571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13108687f45b4b65a798e1c807385416, State: Initialized, Role: FOLLOWER
I20260927 03:22:08.439594 7571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.439677 7421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.439863 7421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6200e0ae8d3e4a7892db9bc23beb195f, State: Initialized, Role: FOLLOWER
I20260927 03:22:08.439885 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.439952 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:08.439929 7571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:08.439981 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.439950 7421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.440222 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.440280 7421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:08.440291 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.441200 7395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6200e0ae8d3e4a7892db9bc23beb195f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404
I20260927 03:22:08.441329 7395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:22:08.441391 7395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 0 FOLLOWER]: Advancing to term 1
W20260927 03:22:08.441887 7596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:08.442005 7596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:08.442278 7395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.442533 7395 leader_election.cc:288] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [CANDIDATE]: Term 1 election: Requested vote from peers d668143132e843a6960eb60f71fb6072 (127.17.52.189:42465), 13108687f45b4b65a798e1c807385416 (127.17.52.188:36847)
W20260927 03:22:08.443885 7603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:08.443944 7603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:08.445070 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13108687f45b4b65a798e1c807385416"
I20260927 03:22:08.445195 7545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:08.446161 7545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6200e0ae8d3e4a7892db9bc23beb195f in term 1.
I20260927 03:22:08.446408 7369 leader_election.cc:302] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13108687f45b4b65a798e1c807385416, 6200e0ae8d3e4a7892db9bc23beb195f; no voters:
I20260927 03:22:08.446563 7580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:08.446707 7580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [term 1 LEADER]: Becoming Leader. State: Replica: 6200e0ae8d3e4a7892db9bc23beb195f, State: Running, Role: LEADER
I20260927 03:22:08.446803 7580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.447554 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6200e0ae8d3e4a7892db9bc23beb195f. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.447665 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:08.441941 7503 master_runner.cc:186] Error getting master registration for 127.17.52.190:35087: OK, instance_id { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" instance_seqno: 1790479328083033 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:22:08.445142 7353 master_runner.cc:186] Error getting master registration for 127.17.52.189:42465: OK, instance_id { permanent_uuid: "d668143132e843a6960eb60f71fb6072" instance_seqno: 1790479328238832 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:22:08.448019 7353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:22:08.448025 7503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:22:08.448194 7614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:08.448428 7614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:08.450047 7545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [term 1 FOLLOWER]: Refusing update from remote peer 6200e0ae8d3e4a7892db9bc23beb195f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:08.450436 7612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:08.451632 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6200e0ae8d3e4a7892db9bc23beb195f. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.451740 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.452819 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.452909 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:08.453327 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.453404 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.453658 7614 catalog_manager.cc:1347] Generated new cluster ID: 9d84187e51c94945bcea7e437ae13561
I20260927 03:22:08.453701 7614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:08.456636 7615 mvcc.cc:205] Tried to move back new op lower bound from 7333803329329221632 to 7333803329320189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333803329329221632 or (T in {7333803329329221632})}]
I20260927 03:22:08.458624 7616 mvcc.cc:205] Tried to move back new op lower bound from 7333803329329221632 to 7333803329320189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333803329329221632}]
I20260927 03:22:08.462836 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.443728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.17.52.189:42465)
0927 03:22:08.443773 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.443776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.444720 (+ 944us) server_negotiation.cc:407] Connection header received
0927 03:22:08.445244 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.445245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.445255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.445283 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.453469 (+ 8186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.454132 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.455988 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.456726 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.456770 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.459766 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.459767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.459767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.459774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.461278 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.462087 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.462089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.462089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.462143 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.462319 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.462320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.462321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.462444 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.462505 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.462677 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.462745 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:22:08.465538 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.444732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.17.52.190:35087)
0927 03:22:08.444776 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.444778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.444975 (+ 197us) server_negotiation.cc:407] Connection header received
0927 03:22:08.445142 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.445143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.445153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.445182 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.452652 (+ 7470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.453369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.457327 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.457617 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.457653 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.461110 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.461111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.461111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.461118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.462659 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.463818 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.463820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.463820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.463879 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.464397 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.464398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.464411 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.464542 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.464653 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.465370 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.465445 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:22:08.469444 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.443970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.17.52.188:36847)
0927 03:22:08.444038 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.444040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.446099 (+ 2059us) server_negotiation.cc:407] Connection header received
0927 03:22:08.446896 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.446897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.446908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.446937 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.455979 (+ 9042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.456747 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.458352 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.458692 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.458731 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.466038 (+ 7307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.466039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.466039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.466046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.467655 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.468296 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.468298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.468298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.468356 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.468696 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.468697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.468697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.468838 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.468920 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.469253 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.469336 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:22:08.471762 7614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:08.472527 7614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:08.472972 7496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } attempt: 3
Sep 27 03:22:08 dist-test-slave-7j43 krb5kdc[7340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479328, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:08.479202 7614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f: Generated new TSK 0
I20260927 03:22:08.479404 7614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:22:08.481225 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.473198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:36847 (local address 127.0.0.1:60838)
0927 03:22:08.473280 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:08.473289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:08.473322 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:08.473465 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:08.473466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:08.473606 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:08.473645 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.473646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.474533 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.474533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.475274 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:08.475275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.475751 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.475751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:08.475803 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.476177 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:08.476181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:08.478260 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:08.480105 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.480106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.480106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.480313 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.480521 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:08.480522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:08.480522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:08.480632 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:08.480915 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:08.480916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:08.480959 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:22:08.481026 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:22:08.481130 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 03:22:08.481658 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:08.473290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.17.52.188:36847)
0927 03:22:08.473340 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.473342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.473352 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:08.473387 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.473388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.473396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.473462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.473715 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.474448 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.475376 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:08.475661 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:08.475709 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:08.478437 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:08.478438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:08.478438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:08.478444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:08.480038 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.480395 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.480397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.480397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.480449 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:08.480690 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:08.480691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:08.480691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:08.480813 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:08.480886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:08.481089 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 03:22:08.481224 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:22:08.483354 7496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: Bootstrap starting.
I20260927 03:22:08.484062 7496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:08.484349 7496 log.cc:824] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:08.485395 7496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: No bootstrap required, opened a new log
I20260927 03:22:08.487378 7496 raft_consensus.cc:348] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.487499 7496 raft_consensus.cc:729] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d668143132e843a6960eb60f71fb6072, State: Initialized, Role: FOLLOWER
I20260927 03:22:08.487571 7496 consensus_queue.cc:260] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } }
I20260927 03:22:08.487864 7496 sys_catalog.cc:564] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:08.488678 7469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:08.489540 7469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [term 1 FOLLOWER]: Refusing update from remote peer 6200e0ae8d3e4a7892db9bc23beb195f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:08.489708 7470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d668143132e843a6960eb60f71fb6072"
I20260927 03:22:08.489836 7612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:08.490415 7470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6200e0ae8d3e4a7892db9bc23beb195f in term 1.
W20260927 03:22:08.490849 7449 tablet.cc:2369] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:22:08.492245 7637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:08.492296 7637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:08.493314 7622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.493455 7622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.493543 7622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6200e0ae8d3e4a7892db9bc23beb195f. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.493638 7622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.493703 7622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:08.493785 7622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:08.494366 7629 mvcc.cc:205] Tried to move back new op lower bound from 7333803329442570240 to 7333803329320189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333803329329221632}]
W20260927 03:22:08.511759 7424 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.510753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.17.52.190:35087)
0927 03:22:08.510795 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.510797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.511122 (+ 325us) server_negotiation.cc:407] Connection header received
0927 03:22:08.511221 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.511222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.511232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.511262 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.511692 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45424: BlockingRecv error: recv got EOF from 127.0.0.1:45424 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:22:08.512856 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.512079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.17.52.189:42465)
0927 03:22:08.512113 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.512115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.512377 (+ 262us) server_negotiation.cc:407] Connection header received
0927 03:22:08.512466 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.512467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.512476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.512505 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.512803 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52244: BlockingRecv error: recv got EOF from 127.0.0.1:52244 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:22:08.513563 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.512266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.17.52.188:36847)
0927 03:22:08.512300 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.512302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.513103 (+ 801us) server_negotiation.cc:407] Connection header received
0927 03:22:08.513193 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.513194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.513203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.513231 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.513509 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60850: BlockingRecv error: recv got EOF from 127.0.0.1:60850 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:22:08.517558 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.516808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.17.52.189:42465)
0927 03:22:08.516840 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.516842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.517048 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:22:08.517149 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.517150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.517165 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.517193 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.517516 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52252: BlockingRecv error: recv got EOF from 127.0.0.1:52252 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:22:08.520311 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.519648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.17.52.189:42465)
0927 03:22:08.519678 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.519680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.519867 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:22:08.519956 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.519957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.519965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.519992 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.520272 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52260: BlockingRecv error: recv got EOF from 127.0.0.1:52260 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:22:08.524019 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.523367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.17.52.189:42465)
0927 03:22:08.523397 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.523398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.523583 (+ 185us) server_negotiation.cc:407] Connection header received
0927 03:22:08.523671 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.523672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.523680 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.523707 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.523979 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52270: BlockingRecv error: recv got EOF from 127.0.0.1:52270 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 03:22:08.529811 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.529063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.17.52.189:42465)
0927 03:22:08.529093 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.529095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.529279 (+ 184us) server_negotiation.cc:407] Connection header received
0927 03:22:08.529410 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.529411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.529421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.529449 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.529757 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52286: BlockingRecv error: recv got EOF from 127.0.0.1:52286 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:22:08.539606 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.538879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.17.52.189:42465)
0927 03:22:08.538912 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.538913 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.539113 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:22:08.539218 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.539219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.539228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.539256 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.539563 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52290: BlockingRecv error: recv got EOF from 127.0.0.1:52290 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:22:08.557701 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.556854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.17.52.189:42465)
0927 03:22:08.556894 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.556896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.557140 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:22:08.557242 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.557243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.557254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.557284 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.557644 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52304: BlockingRecv error: recv got EOF from 127.0.0.1:52304 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:22:08.591943 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.591003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.17.52.189:42465)
0927 03:22:08.591047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.591049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.591314 (+ 265us) server_negotiation.cc:407] Connection header received
0927 03:22:08.591441 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.591442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.591454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.591485 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.591882 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52308: BlockingRecv error: recv got EOF from 127.0.0.1:52308 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:22:08.658175 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.657437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.17.52.189:42465)
0927 03:22:08.657515 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.657517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.657654 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:22:08.657713 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.657714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.657725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.657765 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.658115 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52316: BlockingRecv error: recv got EOF from 127.0.0.1:52316 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 03:22:08.788157 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:08.787326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.17.52.189:42465)
0927 03:22:08.787392 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:08.787394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:08.787497 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:22:08.787543 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:08.787544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:08.787555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:08.787617 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:08.788064 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52328: BlockingRecv error: recv got EOF from 127.0.0.1:52328 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 03:22:09.046162 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:09.045429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.17.52.189:42465)
0927 03:22:09.045498 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.045500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.045511 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:22:09.045594 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.045595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.045613 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.045710 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:09.046095 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52342: BlockingRecv error: recv got EOF from 127.0.0.1:52342 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:22:09.086083 7693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6200e0ae8d3e4a7892db9bc23beb195f" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 35087 } } peers { permanent_uuid: "d668143132e843a6960eb60f71fb6072" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 42465 } } peers { permanent_uuid: "13108687f45b4b65a798e1c807385416" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 36847 } } }
I20260927 03:22:09.086222 7693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6200e0ae8d3e4a7892db9bc23beb195f [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:09.442457 7596 catalog_manager.cc:1259] Loaded cluster ID: 9d84187e51c94945bcea7e437ae13561
I20260927 03:22:09.442534 7596 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: loading cluster ID for follower catalog manager: success
I20260927 03:22:09.443348 7596 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: acquiring CA information for follower catalog manager: success
I20260927 03:22:09.443679 7596 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13108687f45b4b65a798e1c807385416: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:22:09.492766 7637 catalog_manager.cc:1259] Loaded cluster ID: 9d84187e51c94945bcea7e437ae13561
I20260927 03:22:09.492844 7637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: loading cluster ID for follower catalog manager: success
I20260927 03:22:09.493700 7637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: acquiring CA information for follower catalog manager: success
I20260927 03:22:09.494025 7637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d668143132e843a6960eb60f71fb6072: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:22:09.562137 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:09.559278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.17.52.189:42465)
0927 03:22:09.559423 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.559426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.559453 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:22:09.559544 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.559545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.559624 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.559739 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:09.559898 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.560438 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.561271 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.561494 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.561582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:09.561776 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:09.561900 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:09.561991 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:09.562014 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 03:22:09.562038 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260927 03:22:09.566502 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:09.563982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.17.52.188:36847)
0927 03:22:09.564105 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.564107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.564114 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:22:09.564161 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.564162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.564203 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.564291 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:09.564386 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.564947 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.565689 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.565902 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.565977 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:09.566142 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:09.566258 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:09.566341 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:09.566393 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:22:09.566414 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260927 03:22:09.570734 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:09.568007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.17.52.190:35087)
0927 03:22:09.568144 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.568147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.568265 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:22:09.568413 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.568414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.568460 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.568538 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:09.568681 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.569176 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.569913 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.570129 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.570203 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:09.570404 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:09.570500 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:09.570559 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:09.570630 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:22:09.570650 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 03:22:09.571231 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:09.568228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.17.52.188:36847)
0927 03:22:09.568307 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.568309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.568431 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:22:09.568523 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.568524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.568536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.568565 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:09.568758 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.569460 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.570449 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.570719 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.570757 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:09.570946 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:09.570974 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:09.571000 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:09.571108 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 03:22:09.571128 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 03:22:09.571369 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:09.568006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.17.52.189:42465)
0927 03:22:09.568122 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:09.568124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:09.568271 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:22:09.568347 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:09.568348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:09.568360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:09.568412 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:09.568594 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.569337 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.570429 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:09.570726 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:09.570789 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:09.570989 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:09.571024 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:09.571070 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:09.571253 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 03:22:09.571276 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 03:22:09.572728 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7353
I20260927 03:22:09.578224 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7428
I20260927 03:22:09.583165 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7503
2026-09-27T03:22:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2824 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:09 dist-test-slave-7j43 krb5kdc[7724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:09 dist-test-slave-7j43 krb5kdc[7724](info): set up 2 sockets
Sep 27 03:22:09 dist-test-slave-7j43 krb5kdc[7724](info): commencing operation
krb5kdc: starting...
W20260927 03:22:10.621366 17618 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/test-user.keytab.
Sep 27 03:22:10 dist-test-slave-7j43 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.52.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
Entry for principal kudu/127.17.52.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab.
I20260927 03:22:10.736930 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:34541
--webserver_interface=127.17.52.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:10.817310 7737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:10.817595 7737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:10.817626 7737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:10.817672 7737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:10.818868 7737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:10.818938 7737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:10.818961 7737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:10.818984 7737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:10.819012 7737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:10.820544 7737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.190.keytab
--principal=kudu/127.17.52.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.190:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.52.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:10.820863 7737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:10.821099 7737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:10.825937 7743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:10.825961 7745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:10.825930 7742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:10.826210 7737 server_base.cc:1034] running on GCE node
I20260927 03:22:10.826475 7737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:10.826812 7737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:10.828028 7737 hybrid_clock.cc:648] HybridClock initialized: now 1790479330828014 us; error 41 us; skew 500 ppm
Sep 27 03:22:10 dist-test-slave-7j43 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:10.830673 7737 init.cc:377] Logged in from keytab as kudu/127.17.52.190@KRBTEST.COM (short username kudu)
I20260927 03:22:10.831158 7737 webserver.cc:460] Webserver started at http://127.17.52.190:38401/ using document root <none> and password file <none>
I20260927 03:22:10.831422 7737 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:10.831497 7737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:10.831661 7737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:10.832829 7737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data/instance:
uuid: "95778a528e7b46df96d764bf6c663f94"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "ab581764faa803fc10fd95c9893a50ad"
server_key_iv: "88a0ef01a68a074e4ff7c6eb84634139"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.833220 7737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance:
uuid: "95778a528e7b46df96d764bf6c663f94"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "ab581764faa803fc10fd95c9893a50ad"
server_key_iv: "88a0ef01a68a074e4ff7c6eb84634139"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.834631 7737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:10.835379 7752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:10.835549 7737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:10.835656 7737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal
uuid: "95778a528e7b46df96d764bf6c663f94"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "ab581764faa803fc10fd95c9893a50ad"
server_key_iv: "88a0ef01a68a074e4ff7c6eb84634139"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.835762 7737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:10.860884 7737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:10.862910 7737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:10.863081 7737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:10.867048 7737 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.190:34541
I20260927 03:22:10.867087 7804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.190:34541 every 8 connection(s)
I20260927 03:22:10.867453 7737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:10.868448 7805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:10.870018 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:10.871999 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7737
I20260927 03:22:10.872099 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-0/wal/instance
I20260927 03:22:10.872289 17618 external_mini_cluster.cc:1428] Setting key 81723d4ed08229d63ad7bfe3a3107a87
Sep 27 03:22:10 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.189@KRBTEST.COM; defaulting to no policy
I20260927 03:22:10.880664 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.870746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.17.52.190:34541)
0927 03:22:10.870932 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:10.870936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:10.870979 (+ 43us) server_negotiation.cc:407] Connection header received
0927 03:22:10.871522 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:10.871524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:10.871543 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:10.871661 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:10.872395 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:10.873304 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:10.874260 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:10.874799 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:10.874918 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:10.877494 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:10.877500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:10.877501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:10.877507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:10.879031 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:10.879499 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:10.879501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:10.879501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:10.879571 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:10.879847 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:10.879848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:10.879848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:10.880036 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:10.880095 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:10.880362 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:22:10.880451 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260927 03:22:10.880782 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.870695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34541 (local address 127.0.0.1:48908)
0927 03:22:10.870868 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:10.870912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:10.871354 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:10.871658 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:10.871660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:10.872026 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:10.872281 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:10.872287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:10.873405 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:10.873406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:10.874171 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:10.874173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:10.874917 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:10.874918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:10.874970 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:10.875494 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:10.875502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:10.877405 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:10.879123 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:10.879130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:10.879132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:10.879413 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:10.879651 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:10.879652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:10.879652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:10.879774 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:10.880114 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:10.880123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:10.880205 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 03:22:10.880323 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:22:10.880593 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260927 03:22:10.881596 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:10.881958 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.881830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52364)
0927 03:22:10.881884 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:10.881938 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:22:10.882109 7754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
I20260927 03:22:10.882506 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.882410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52374)
0927 03:22:10.882459 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:10.882482 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:22:10.882632 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37129: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
Entry for principal kudu/127.17.52.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab.
I20260927 03:22:10.899062 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:37129
--webserver_interface=127.17.52.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:22:10.932004 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } attempt: 1
I20260927 03:22:10.932550 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.932375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52376)
0927 03:22:10.932454 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:10.932508 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:22:10.932989 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:10.932896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52386)
0927 03:22:10.932941 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:10.932970 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:22:10.933117 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37129: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
W20260927 03:22:10.980086 7812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:10.980297 7812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:10.980314 7812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:10.980334 7812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:10.981513 7812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:10.981552 7812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:10.981566 7812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:10.981613 7812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:10.981639 7812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:10.983088 7812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.189.keytab
--principal=kudu/127.17.52.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.189:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.52.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:10.983355 7812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:10.983587 7812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:10.988196 7818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:10.988205 7820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:10.988270 7817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:10.988425 7812 server_base.cc:1034] running on GCE node
I20260927 03:22:10.988768 7812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:10.989001 7812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:10.990178 7812 hybrid_clock.cc:648] HybridClock initialized: now 1790479330990161 us; error 33 us; skew 500 ppm
Sep 27 03:22:10 dist-test-slave-7j43 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:10.992748 7812 init.cc:377] Logged in from keytab as kudu/127.17.52.189@KRBTEST.COM (short username kudu)
I20260927 03:22:10.993175 7812 webserver.cc:460] Webserver started at http://127.17.52.189:46079/ using document root <none> and password file <none>
I20260927 03:22:10.993369 7812 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:10.993408 7812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:10.993499 7812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:10.994441 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data/instance:
uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "10a217912b3b720d455adc7fd75a59dd"
server_key_iv: "c2ca37765aafd96a6f5db963b3858216"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.994738 7812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal/instance:
uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "10a217912b3b720d455adc7fd75a59dd"
server_key_iv: "c2ca37765aafd96a6f5db963b3858216"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.995913 7812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:10.996605 7827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:10.996803 7812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:10.996893 7812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal
uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8"
format_stamp: "Formatted at 2026-09-27 03:22:10 on dist-test-slave-7j43"
server_key: "10a217912b3b720d455adc7fd75a59dd"
server_key_iv: "c2ca37765aafd96a6f5db963b3858216"
server_key_version: "encryptionkey@0"
I20260927 03:22:10.996984 7812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:11.001268 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } attempt: 2
I20260927 03:22:11.001652 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.001521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52396)
0927 03:22:11.001581 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.001618 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:22:11.001994 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.001907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52412)
0927 03:22:11.001949 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.001974 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:22:11.002100 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.189:37129: Network error: Client connection negotiation failed: client connection to 127.17.52.189:37129: connect: Connection refused (error 111)
I20260927 03:22:11.024952 7812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:11.026880 7812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:11.027035 7812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:11.030958 7812 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.189:37129
I20260927 03:22:11.030992 7879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.189:37129 every 8 connection(s)
I20260927 03:22:11.031355 7812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/data/info.pb
I20260927 03:22:11.032346 7880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:11.033869 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7812
I20260927 03:22:11.033941 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-1/wal/instance
I20260927 03:22:11.034135 17618 external_mini_cluster.cc:1428] Setting key 3a883dbb011158276f70f655fd7073f7
I20260927 03:22:11.033906 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.52.188@KRBTEST.COM; defaulting to no policy
I20260927 03:22:11.043430 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.034666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.17.52.190:34541)
0927 03:22:11.034720 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.034722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.034946 (+ 224us) server_negotiation.cc:407] Connection header received
0927 03:22:11.035436 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.035438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.035451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.035513 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.036123 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.036685 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.037737 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.037995 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.038073 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.040533 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.040534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.040534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.040540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.042026 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.042439 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.042440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.042441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.042501 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.042772 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.042773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.042773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.042885 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.042962 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.043222 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.043341 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:22:11.043584 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.034647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34541 (local address 127.0.0.1:48916)
0927 03:22:11.034823 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.034864 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.035325 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.035589 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.035592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.035845 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.036028 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.036032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.036790 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.036791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.037651 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.037653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.038092 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.038093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.038216 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.038701 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.038709 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.040443 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.042098 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.042104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.042106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.042352 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.042575 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.042576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.042577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.042677 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.042952 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.042959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.043088 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.043202 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.043361 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.52.188@KRBTEST.COM" created.
I20260927 03:22:11.044180 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:11.051604 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.044478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52422)
0927 03:22:11.044640 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.044649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.044711 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.044902 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.044902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.045037 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.045076 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.045078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.045916 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.045916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.046404 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.046406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.046949 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.046949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.046992 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.047379 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.047384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.048781 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.050529 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.050529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.050530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.050694 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.050916 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.050916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.050917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.050970 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.051277 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.051278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.051315 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.051382 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.051486 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 03:22:11.051671 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.044547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.17.52.189:37129)
0927 03:22:11.044739 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.044744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.044767 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:22:11.044786 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.044788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.044799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.044903 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.045157 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.045808 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.046493 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.046793 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.046883 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.048886 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.048893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.048894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.048900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.050426 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.050788 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.050791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.050792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.050850 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.051059 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.051059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.051060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.051208 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.051308 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.051410 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.051539 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":164,"thread_start_us":120,"threads_started":1}
I20260927 03:22:11.052409 7880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:11.052856 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.052701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55288)
0927 03:22:11.052771 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.052816 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 03:22:11.053042 7831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
I20260927 03:22:11.053442 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.053383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55290)
0927 03:22:11.053406 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.053427 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 03:22:11.053516 7880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:33069: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
Entry for principal kudu/127.17.52.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab.
I20260927 03:22:11.058772 17618 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRLekJh/build/release/bin/kudu
/tmp/dist-test-taskRLekJh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:33069
--webserver_interface=127.17.52.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.52.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:22:11.093237 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } attempt: 3
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:11.100832 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.093738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.17.52.189:37129)
0927 03:22:11.093804 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.093806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.093815 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:22:11.093846 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.093851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.093863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.093931 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.094260 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.094757 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.095472 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.095690 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.095776 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.098003 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.098005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.098005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.098013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.099456 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.099956 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.099958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.099958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.100029 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.100245 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.100246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.100247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.100383 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.100458 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.100665 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.100743 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:22:11.100867 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.093642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52428)
0927 03:22:11.093707 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.093716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.093752 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.093938 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.093939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.094109 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.094173 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.094175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.094849 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.094851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.095386 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.095388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.095773 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.095774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.095864 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.096320 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.096332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.097899 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.099560 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.099562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.099562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.099781 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.100104 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.100105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.100105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.100178 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.100460 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.100461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.100508 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.100618 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.100728 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:22:11.101212 7805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:22:11.101588 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.101486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55302)
0927 03:22:11.101538 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.101568 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:22:11.102274 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.102180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55304)
0927 03:22:11.102230 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.102257 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:22:11.102389 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:33069: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
I20260927 03:22:11.102725 7880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } attempt: 1
I20260927 03:22:11.103060 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.102981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55318)
0927 03:22:11.103014 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.103041 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:22:11.103323 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.103269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55320)
0927 03:22:11.103289 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.103310 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 03:22:11.103422 7880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:33069: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
I20260927 03:22:11.133528 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } attempt: 1
I20260927 03:22:11.134099 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.133921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55322)
0927 03:22:11.134027 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.134069 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 03:22:11.134497 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.134414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55336)
0927 03:22:11.134457 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.134481 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:22:11.134606 7805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:33069: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
W20260927 03:22:11.142701 7887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:22:11.142891 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:11.142907 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:11.142957 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:11.144079 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:11.144119 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:11.144133 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:11.144145 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:11.144201 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:11.145715 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.52.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.52.190:34541,127.17.52.189:37129,127.17.52.188:33069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/krb5kdc/kudu_127.17.52.188.keytab
--principal=kudu/127.17.52.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.52.188:33069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.52.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:11.145980 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:11.146220 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:11.150959 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:11.150965 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:11.150964 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:11.151253 7887 server_base.cc:1034] running on GCE node
I20260927 03:22:11.151502 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:11.151854 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:11.153018 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790479331153007 us; error 42 us; skew 500 ppm
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479331, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:11.155444 7887 init.cc:377] Logged in from keytab as kudu/127.17.52.188@KRBTEST.COM (short username kudu)
I20260927 03:22:11.155929 7887 webserver.cc:460] Webserver started at http://127.17.52.188:45237/ using document root <none> and password file <none>
I20260927 03:22:11.156149 7887 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:11.156215 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:11.156368 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:11.157460 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data/instance:
uuid: "588b5334dc414f47b14b5f7a89e5c2a4"
format_stamp: "Formatted at 2026-09-27 03:22:11 on dist-test-slave-7j43"
server_key: "4ba9caf13f4e0896cc31070965dd0d11"
server_key_iv: "f19d1489e15650afaafb021b4fe9f7cf"
server_key_version: "encryptionkey@0"
I20260927 03:22:11.157828 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal/instance:
uuid: "588b5334dc414f47b14b5f7a89e5c2a4"
format_stamp: "Formatted at 2026-09-27 03:22:11 on dist-test-slave-7j43"
server_key: "4ba9caf13f4e0896cc31070965dd0d11"
server_key_iv: "f19d1489e15650afaafb021b4fe9f7cf"
server_key_version: "encryptionkey@0"
I20260927 03:22:11.159122 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:11.159827 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:11.159999 7887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:11.160096 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data,/tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal
uuid: "588b5334dc414f47b14b5f7a89e5c2a4"
format_stamp: "Formatted at 2026-09-27 03:22:11 on dist-test-slave-7j43"
server_key: "4ba9caf13f4e0896cc31070965dd0d11"
server_key_iv: "f19d1489e15650afaafb021b4fe9f7cf"
server_key_version: "encryptionkey@0"
I20260927 03:22:11.160181 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:11.166746 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:11.168370 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:11.168522 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:11.171568 7880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } attempt: 2
I20260927 03:22:11.172011 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.171886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55338)
0927 03:22:11.171941 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.171971 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:22:11.172416 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.172313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55344)
0927 03:22:11.172362 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.172391 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:22:11.172513 7880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.52.188:33069: Network error: Client connection negotiation failed: client connection to 127.17.52.188:33069: connect: Connection refused (error 111)
I20260927 03:22:11.172592 7887 rpc_server.cc:307] RPC server started. Bound to: 127.17.52.188:33069
I20260927 03:22:11.172633 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.52.188:33069 every 8 connection(s)
I20260927 03:22:11.172960 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/data/info.pb
I20260927 03:22:11.173667 17618 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRLekJh/build/release/bin/kudu as pid 7887
I20260927 03:22:11.173772 17618 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRLekJh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479066103559-17618-0/minicluster-data/master-2/wal/instance
I20260927 03:22:11.174010 17618 external_mini_cluster.cc:1428] Setting key 6183e0db156422bce61b2d234ff7273b
I20260927 03:22:11.174136 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:11.176501 7955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479331, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
I20260927 03:22:11.184639 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.174509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.17.52.188:33069)
0927 03:22:11.174670 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.174679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.174711 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:22:11.175216 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.175222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.175269 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.175377 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.176156 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.177293 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.178316 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.178933 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.179006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.181448 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.181453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.181454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.181459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.182970 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.183314 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.183319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.183319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.183395 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.183597 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.183598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.183598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.183920 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.184040 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.184228 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.184357 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260927 03:22:11.184921 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.177138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.190:34541 (local address 127.0.0.1:48926)
0927 03:22:11.177316 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.177368 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.177438 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.177586 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.177589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.177921 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.177992 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.177994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.178864 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.178864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.179778 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.179782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.180189 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.180190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.180263 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.180823 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.180831 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.182170 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.183668 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.183672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.183674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.183887 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.184170 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.184171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.184171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.184232 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.184530 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.184538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.184615 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.184700 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.184815 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"thread_start_us":73,"threads_started":1}
I20260927 03:22:11.184926 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.177194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.17.52.190:34541)
0927 03:22:11.177249 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.177251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.177450 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:22:11.177494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.177495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.177505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.177548 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.178096 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.178768 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.179880 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.180072 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.180137 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.182232 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.182234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.182234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.182242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.183599 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.184009 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.184011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.184011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.184096 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.184306 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.184307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.184308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.184444 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.184524 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.184763 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.184837 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:22:11.185648 7955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.190@KRBTEST.COM
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479331, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:11.192786 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.185468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.17.52.190:34541)
0927 03:22:11.185504 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.185506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.185560 (+ 54us) server_negotiation.cc:407] Connection header received
0927 03:22:11.185627 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.185628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.185636 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.185692 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.186218 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.186656 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.187564 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.187843 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.187910 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.190267 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.190269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.190269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.190276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.191610 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.191968 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.191968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.191969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.192014 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.192210 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.192210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.192210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.192362 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.192521 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.192565 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.192679 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:11.193364 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.185870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.189:37129 (local address 127.0.0.1:52430)
0927 03:22:11.186233 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.186242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.186301 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.186419 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.186420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.186550 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.186576 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.186577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.187454 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.187455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.188159 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.188162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.188680 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.188681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.188724 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.189086 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.189091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.190488 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.192282 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.192283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.192283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.192476 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.192697 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.192698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.192698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.192755 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.193042 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.193042 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.193102 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.193176 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.193263 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348}
I20260927 03:22:11.193392 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.185940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.17.52.189:37129)
0927 03:22:11.185985 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.185987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.186291 (+ 304us) server_negotiation.cc:407] Connection header received
0927 03:22:11.186352 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.186354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.186363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.186408 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.186659 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.187376 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.188239 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.188567 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.188674 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.190550 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.190551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.190552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.190559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.192210 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.192548 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.192549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.192550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.192627 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.192823 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.192824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.192824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.192971 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.193041 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.193219 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.193300 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:11.193718 7955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479331, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.188@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.52.189@KRBTEST.COM
I20260927 03:22:11.200109 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.193965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55362)
0927 03:22:11.194001 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.194009 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.194039 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.194138 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.194138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.194235 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.194258 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.194259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.195023 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.195024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.195600 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.195601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.196058 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.196058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.196108 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.196461 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.196465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.197593 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.199086 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.199087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.199088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.199298 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.199499 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.199500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.199500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.199568 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.199868 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.199869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.199913 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.199982 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.200050 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:22:11.200464 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.194007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.17.52.188:33069)
0927 03:22:11.194043 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.194045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.194052 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:22:11.194094 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.194094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.194101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.194134 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.194330 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.194953 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.195673 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.195988 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.196050 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.197673 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.197675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.197675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.197683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.199016 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.199367 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.199368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.199368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.199425 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.199650 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.199652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.199652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.199790 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.199863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.200043 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.200111 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:22:11.201604 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.193547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.17.52.189:37129)
0927 03:22:11.193654 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.193656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.193899 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:22:11.193925 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.193926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.193933 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.193995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.194706 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.195261 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.196056 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.196328 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.196409 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.198808 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.198809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.198809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.198816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.200466 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.200944 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.200945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.200945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.200996 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.201197 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.201198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.201198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.201293 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.201407 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.201492 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.201547 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":37}
I20260927 03:22:11.202263 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: Bootstrap starting.
I20260927 03:22:11.203161 17618 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:22:11.203338 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:11.203632 7955 log.cc:824] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:11.204481 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: No bootstrap required, opened a new log
I20260927 03:22:11.204172 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95778a528e7b46df96d764bf6c663f94"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48940
I20260927 03:22:11.206450 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.206599 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588b5334dc414f47b14b5f7a89e5c2a4, State: Initialized, Role: FOLLOWER
I20260927 03:22:11.206674 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.206981 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:11.206946 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.207060 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:22:11.208766 7972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:11.208825 7972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:11.209765 7805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } attempt: 2
I20260927 03:22:11.209004 7887 master_runner.cc:186] Error getting master registration for 127.17.52.190:34541: OK, instance_id { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" instance_seqno: 1790479330865949 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:22:11.211651 7887 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.190@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:11.217002 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.210126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.17.52.188:33069)
0927 03:22:11.210162 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.210164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.210174 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:11.210248 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.210250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.210259 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.210300 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.210639 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.211330 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.212149 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.212339 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.212423 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.214201 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.214203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.214203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.214217 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.215778 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.216156 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.216157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.216157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.216227 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.216427 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.216428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.216428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.216624 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.216697 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.216846 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.216922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:22:11.217003 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.210019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55376)
0927 03:22:11.210121 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.210132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.210177 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.210344 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.210345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.210516 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.210561 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.210563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.211432 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.211433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.212046 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.212048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.212423 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.212424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.212482 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.212961 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.212966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.214120 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.215866 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.215867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.215868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.216082 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.216299 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.216300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.216300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.216361 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.216694 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.216695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.216737 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.216805 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.216904 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 03:22:11.218490 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: Bootstrap starting.
I20260927 03:22:11.219131 7805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:11.219413 7805 log.cc:824] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:11.220142 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: No bootstrap required, opened a new log
I20260927 03:22:11.221174 7805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.221306 7805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95778a528e7b46df96d764bf6c663f94, State: Initialized, Role: FOLLOWER
I20260927 03:22:11.221408 7805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.221691 7805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:11.221813 7779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:22:11.221887 7779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:11.222656 7779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.222900 7779 leader_election.cc:288] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [CANDIDATE]: Term 1 election: Requested vote from peers 61684aea90ac4f7b92fb3cc4a8cba6b8 (127.17.52.189:37129), 588b5334dc414f47b14b5f7a89e5c2a4 (127.17.52.188:33069)
I20260927 03:22:11.223234 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95778a528e7b46df96d764bf6c663f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "588b5334dc414f47b14b5f7a89e5c2a4"
I20260927 03:22:11.223361 7929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:11.223845 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.223932 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.224262 7929 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95778a528e7b46df96d764bf6c663f94 in term 1.
I20260927 03:22:11.224738 7756 leader_election.cc:302] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588b5334dc414f47b14b5f7a89e5c2a4, 95778a528e7b46df96d764bf6c663f94; no voters:
I20260927 03:22:11.224861 7975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:11.224989 7975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [term 1 LEADER]: Becoming Leader. State: Replica: 95778a528e7b46df96d764bf6c663f94, State: Running, Role: LEADER
I20260927 03:22:11.225070 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.225546 7992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95778a528e7b46df96d764bf6c663f94. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.225761 7992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:11.224316 7737 master_runner.cc:186] Error getting master registration for 127.17.52.189:37129: OK, instance_id { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" instance_seqno: 1790479331029885 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:22:11.226598 7737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:22:11.227139 7986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:11.227190 7986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:11.227375 7995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:11.227540 7995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:11.228978 7929 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [term 1 FOLLOWER]: Refusing update from remote peer 95778a528e7b46df96d764bf6c663f94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:11.229265 7975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:11.230533 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95778a528e7b46df96d764bf6c663f94. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.230621 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.234673 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.224157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.17.52.188:33069)
0927 03:22:11.224192 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.224194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.226088 (+ 1894us) server_negotiation.cc:407] Connection header received
0927 03:22:11.226183 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.226184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.226193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.226232 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.226971 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.227710 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.228796 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.229094 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.229133 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.231415 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.231417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.231417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.231424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.233160 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.233814 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.233815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.233816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.233870 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.234063 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.234064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.234064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.234181 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.234261 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.234497 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.234576 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:11.237753 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.237850 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:11.238579 7998 mvcc.cc:205] Tried to move back new op lower bound from 7333803340711673856 to 7333803340698656768. Current Snapshot: MvccSnapshot[applied={T|T < 7333803340711673856}]
I20260927 03:22:11.239982 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.240072 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.240921 7999 mvcc.cc:205] Tried to move back new op lower bound from 7333803340711673856 to 7333803340698656768. Current Snapshot: MvccSnapshot[applied={T|T < 7333803340711673856}]
I20260927 03:22:11.241111 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.225638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.17.52.190:34541)
0927 03:22:11.225672 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.225673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.232766 (+ 7093us) server_negotiation.cc:407] Connection header received
0927 03:22:11.232865 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.232866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.232877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.232907 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.233521 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.234252 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.235829 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.236128 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.236165 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.237957 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.237958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.237958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.237965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.239687 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.240263 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.240264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.240265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.240319 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.240512 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.240513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.240514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.240665 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.240738 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.240945 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.241022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:22:11.245072 7995 catalog_manager.cc:1347] Generated new cluster ID: 962b3c04b5304c0d8b7d5e5cfdbed3d0
I20260927 03:22:11.245131 7995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:11.248606 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.226721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.17.52.189:37129)
0927 03:22:11.226784 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.226786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.234781 (+ 7995us) server_negotiation.cc:407] Connection header received
0927 03:22:11.234811 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.234812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.234824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.234878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.235484 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.236183 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.238345 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.239201 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.239261 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.245532 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.245534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.245534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.245557 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.247368 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.247741 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.247742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.247743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.247799 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.248016 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.248017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.248017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.248138 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.248218 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.248406 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.248483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:22:11.263638 7880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } attempt: 3
Sep 27 03:22:11 dist-test-slave-7j43 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479330, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.52.189@KRBTEST.COM for kudu/127.17.52.188@KRBTEST.COM
I20260927 03:22:11.271737 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.263894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.52.188:33069 (local address 127.0.0.1:55400)
0927 03:22:11.263934 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:11.263945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:11.263986 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:11.264192 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:11.264193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:11.264379 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:11.264433 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.264435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.265363 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.265363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.266167 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:11.266169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.266712 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.266713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:11.266771 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.267241 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:11.267246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:11.268729 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:11.270707 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.270708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.270709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.270935 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.271122 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:11.271123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:11.271123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:11.271181 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:11.271431 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:11.271431 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:11.271477 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:22:11.271544 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:22:11.271635 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:22:11.272445 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:11.263983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.17.52.188:33069)
0927 03:22:11.264087 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.264089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.264096 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:22:11.264118 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.264119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.264128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.264189 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.264513 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.265275 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.266282 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:11.266603 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:11.266660 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:11.268805 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:11.268806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:11.268806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:11.268813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:11.270626 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.271004 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.271006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.271006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.271060 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:11.271249 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:11.271250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:11.271251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:11.271377 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:11.271438 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:11.272262 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 03:22:11.272353 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 03:22:11.274282 7880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: Bootstrap starting.
I20260927 03:22:11.275027 7880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:11.275308 7880 log.cc:824] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:11.280808 7995 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:11.281260 7880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: No bootstrap required, opened a new log
I20260927 03:22:11.282243 7995 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:11.282845 7880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.282979 7880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61684aea90ac4f7b92fb3cc4a8cba6b8, State: Initialized, Role: FOLLOWER
I20260927 03:22:11.283071 7880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } }
I20260927 03:22:11.283387 8005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.283460 7880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:11.283565 7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95778a528e7b46df96d764bf6c663f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8"
I20260927 03:22:11.283686 7854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:11.284548 7854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95778a528e7b46df96d764bf6c663f94 in term 1.
I20260927 03:22:11.285514 7853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [term 1 FOLLOWER]: Refusing update from remote peer 95778a528e7b46df96d764bf6c663f94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:11.284878 8005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.285907 7992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:22:11.289400 8016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:22:11.289497 8016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:22:11.289598 8005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95778a528e7b46df96d764bf6c663f94. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.289700 8005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.289772 8005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.289863 8005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:22:11.290529 8017 mvcc.cc:205] Tried to move back new op lower bound from 7333803340920209408 to 7333803340698656768. Current Snapshot: MvccSnapshot[applied={T|T < 7333803340711673856}]
I20260927 03:22:11.301021 8002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95778a528e7b46df96d764bf6c663f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95778a528e7b46df96d764bf6c663f94" member_type: VOTER last_known_addr { host: "127.17.52.190" port: 34541 } } peers { permanent_uuid: "61684aea90ac4f7b92fb3cc4a8cba6b8" member_type: VOTER last_known_addr { host: "127.17.52.189" port: 37129 } } peers { permanent_uuid: "588b5334dc414f47b14b5f7a89e5c2a4" member_type: VOTER last_known_addr { host: "127.17.52.188" port: 33069 } } }
I20260927 03:22:11.301141 8002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:11.304648 7995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95778a528e7b46df96d764bf6c663f94: Generated new TSK 0
I20260927 03:22:11.304870 7995 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:22:11.333839 7958 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.332128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.17.52.188:33069)
0927 03:22:11.332173 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.332175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.333172 (+ 997us) server_negotiation.cc:407] Connection header received
0927 03:22:11.333278 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.333280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.333290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.333320 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.333774 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55404: BlockingRecv error: recv got EOF from 127.0.0.1:55404 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:22:11.334744 7808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.332530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.17.52.190:34541)
0927 03:22:11.332607 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.332608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.334220 (+ 1612us) server_negotiation.cc:407] Connection header received
0927 03:22:11.334316 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.334317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.334326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.334368 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.334684 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:22:11.335734 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.332896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.17.52.189:37129)
0927 03:22:11.332936 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.332938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.335084 (+ 2146us) server_negotiation.cc:407] Connection header received
0927 03:22:11.335269 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.335271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.335282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.335314 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.335674 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52458: BlockingRecv error: recv got EOF from 127.0.0.1:52458 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:22:11.338228 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.337420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.17.52.189:37129)
0927 03:22:11.337454 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.337456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.337678 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:22:11.337782 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.337783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.337809 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.337839 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.338184 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52468: BlockingRecv error: recv got EOF from 127.0.0.1:52468 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:22:11.341364 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.340629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.17.52.189:37129)
0927 03:22:11.340663 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.340665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.340887 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:22:11.340978 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.340979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.340989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.341017 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.341308 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52472: BlockingRecv error: recv got EOF from 127.0.0.1:52472 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:22:11.345283 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.344519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.17.52.189:37129)
0927 03:22:11.344551 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.344552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.344810 (+ 258us) server_negotiation.cc:407] Connection header received
0927 03:22:11.344905 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.344906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.344915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.344944 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.345243 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52474: BlockingRecv error: recv got EOF from 127.0.0.1:52474 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:22:11.351254 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.350478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.17.52.189:37129)
0927 03:22:11.350512 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.350513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.350745 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:22:11.350842 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.350843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.350854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.350883 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.351210 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52482: BlockingRecv error: recv got EOF from 127.0.0.1:52482 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:22:11.361486 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.360617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.17.52.189:37129)
0927 03:22:11.360659 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.360661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.360919 (+ 258us) server_negotiation.cc:407] Connection header received
0927 03:22:11.361022 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.361023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.361034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.361063 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.361431 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52492: BlockingRecv error: recv got EOF from 127.0.0.1:52492 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:22:11.379940 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.379003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.17.52.189:37129)
0927 03:22:11.379047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.379049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.379325 (+ 276us) server_negotiation.cc:407] Connection header received
0927 03:22:11.379451 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.379453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.379465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.379495 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.379880 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52504: BlockingRecv error: recv got EOF from 127.0.0.1:52504 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:22:11.414112 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.413424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.17.52.189:37129)
0927 03:22:11.413480 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.413482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.413608 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:22:11.413640 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.413642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.413655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.413724 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.414051 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52506: BlockingRecv error: recv got EOF from 127.0.0.1:52506 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 03:22:11.480027 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.479297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.17.52.189:37129)
0927 03:22:11.479389 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.479391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.479405 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:22:11.479487 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.479488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.479499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.479551 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.479942 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52514: BlockingRecv error: recv got EOF from 127.0.0.1:52514 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 03:22:11.610095 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.609353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.17.52.189:37129)
0927 03:22:11.609450 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.609452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.609529 (+ 77us) server_negotiation.cc:407] Connection header received
0927 03:22:11.609590 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.609591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.609602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.609667 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.610002 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52516: BlockingRecv error: recv got EOF from 127.0.0.1:52516 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 03:22:11.868330 7884 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:22:11.867565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.17.52.189:37129)
0927 03:22:11.867629 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:11.867635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:11.867666 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:22:11.867691 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:11.867692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:11.867702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:11.867769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:11.868269 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52524: BlockingRecv error: recv got EOF from 127.0.0.1:52524 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:22:12.209357 7972 catalog_manager.cc:1259] Loaded cluster ID: 962b3c04b5304c0d8b7d5e5cfdbed3d0
I20260927 03:22:12.209481 7972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: loading cluster ID for follower catalog manager: success
I20260927 03:22:12.210289 7972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: acquiring CA information for follower catalog manager: success
I20260927 03:22:12.210599 7972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 588b5334dc414f47b14b5f7a89e5c2a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:22:12.289991 8016 catalog_manager.cc:1259] Loaded cluster ID: 962b3c04b5304c0d8b7d5e5cfdbed3d0
I20260927 03:22:12.290071 8016 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: loading cluster ID for follower catalog manager: success
I20260927 03:22:12.290876 8016 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: acquiring CA information for follower catalog manager: success
I20260927 03:22:12.291131 8016 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61684aea90ac4f7b92fb3cc4a8cba6b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:22:12.384047 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:12.381571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.17.52.189:37129)
0927 03:22:12.381701 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:12.381703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:12.381715 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:22:12.381796 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:12.381797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:12.381847 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:12.381932 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:12.382065 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.382578 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.383300 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.383560 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.383612 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:12.383755 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:12.383864 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:12.383924 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:12.383961 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:22:12.383979 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260927 03:22:12.388237 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:12.385933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.17.52.188:33069)
0927 03:22:12.386055 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:12.386058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:12.386067 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:22:12.386147 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:12.386148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:12.386195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:12.386298 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:12.386405 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.386878 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.387560 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.387746 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.387792 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:12.387914 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:12.388010 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:12.388065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:12.388099 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:22:12.388115 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260927 03:22:12.392745 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:12.389854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.17.52.190:34541)
0927 03:22:12.390051 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:12.390053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:12.390302 (+ 249us) server_negotiation.cc:407] Connection header received
0927 03:22:12.390352 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:12.390353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:12.390399 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:12.390492 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:12.390606 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.391104 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.391931 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.392164 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.392238 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:12.392412 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:12.392490 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:12.392537 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:12.392643 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:22:12.392665 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":51,"threads_started":1}
I20260927 03:22:12.393481 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:12.389855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.17.52.189:37129)
0927 03:22:12.389941 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:12.389943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:12.390099 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:22:12.390167 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:12.390168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:12.390180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:12.390230 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:12.390355 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.391126 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.392304 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.392567 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.392648 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:12.393221 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:12.393250 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:12.393320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:12.393378 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:22:12.393402 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 03:22:12.393549 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:12.389903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.17.52.188:33069)
0927 03:22:12.389966 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:12.389967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:12.390149 (+ 182us) server_negotiation.cc:407] Connection header received
0927 03:22:12.390210 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:12.390211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:12.390220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:12.390267 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:12.390460 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.391148 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.392643 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:12.392923 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:12.392984 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:12.393121 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:12.393156 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:12.393206 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:12.393437 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:22:12.393462 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:22:12.394697 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7737
I20260927 03:22:12.400204 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7812
I20260927 03:22:12.404999 17618 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRLekJh/build/release/bin/kudu with pid 7887
2026-09-27T03:22:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2820 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5645 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266308 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST